WO2025022401A2 - Interactive production portfolio generation - Google Patents

Abstract

Methods and systems are provided for generating an interactive production portfolio for manufacturing and/or maintenance of a product, including generation of a verified production knowledge base that includes a project catalog and additional digital engineering procedures and documents, while guiding a user to create: assembly procedures, testing procedures, disassembly procedures, rework procedures, maintenance procedures, fault isolation procedures, calibration procedures, operating procedures, packaging procedures, testing procedures, acceptance tests procedures and other engineering procedures. The portfolio includes a graph data structure enabling reuse and sharing of components and a hierarchical tree of parts, each part linked to one or more models, illustrations, and/or linking balloon indices.

Description

INTERACTIVE PRODUCTION PORTFOLIO GENERATION

FIELD OF THE INVENTION

[0001] The present invention relates to the field of interactive techniques for managing and validating manufacturing and maintenance processes.

BACKGROUND

[0002] In a modem manufacturing organization, data regarding the manufacture of parts/products must be available to people throughout the organization. Accountants, for example, need bills of materials (BOMs) to budget the manufacturing costs for products. Engineers need a more detailed engineering bill of material (EBOM) that includes not only the parts and subassemblies necessary to build products, but also more detailed information, such as tolerances, standards, and specifications. A further in-depth version of the EBOM, also referred to as a manufacturing bill of materials (MBOM) may be available for use in the actual manufacturing operations and may include descriptions of processing steps, required machinery, and packaging. A further type of BOM, also referred to as a service BOM (SBOM) may include details on maintenance requirements related to product sub-assemblies. [0003] While a large set of procedure files must be produced for a production portfolio during the life-cycle of project, those documents are frequently incomplete, inaccurate, outdated or missing; leading to lower yields, recurrent rework, longer time to market and significant added costs. A well-known cause for the inaccuracy of the documentation is the complexity of coordinating communication between the R&D, engineering, and production divisions. However, a deeper cause is the lack of detailed methods for transforming a 2/3D model to a verifiable production file. The present invention describes manual and automatic methods for closing the engineering gaps. SUMMARY

[0004] Embodiments of the present invention provide a system and methods for generating an interactive production portfolio. A production portfolio includes an illustrated parts catalog, an EBOM, mechanical and electrical drawings and models, and procedures for assembly, disassembly, rework, faults isolation, testing, tutorials, maintenance, packing, installation, and operation.

BRIEF DESCRIPTION OF DRAWINGS

[0005] For a better understanding of various embodiments of the invention and to show how the same may be carried into effect, reference is made hereinbelow, by way of example, to the accompanying drawings. Structural details of the invention are shown to provide a fundamental understanding of the invention, the description, taken with the drawings, making apparent to those skilled in the art how the several forms of the invention may be embodied in practice. In the accompanying drawings:

[0006] Fig. 1 is a schematic, flow diagram of a workflow for generating a standardized, verifiable, interactive project portfolio, according to embodiments of the present invention;

[0007] Fig. 2 is a schematic class diagram of classes of objects stored in the interactive project portfolio, according to embodiments of the present invention;

[0008] Fig. 3 is an example of part hierarchies of two similar projects sharing multiple assemblies, according to embodiments of the present invention;

[0009] Fig. 4 shows a flow of steps (FOS) of an assembly procedure and the effect of the assembly FOS in linking a tree of process parts, each linked to an assembly and its subparts, according to embodiments of the present invention;

[0010] Fig 5 shows an exemplary presentation of a BOM image associated with a given step of a FOS; and [0011] Figs. 6-7 show an assembly and job rework process, including execution of alternative steps for invalid results, which might be implemented based on a FOS of the portfolio.

DETAILED DESCRIPTION

[0012] Examples of the present invention provide a system and method that generates standardized, verifiable assembly procedures of an on-line, interactive project portfolio, thereby providing an up-to-date knowledge base of manufacturing information, accessible in real time to all users in a manufacturing environment. Engineering knowledge, also referred to herein as “content,” or “manufacturing documentation,” includes a wide variety of instructions related to processes extending from manufacturing to maintenance, including but not limited to safety, assembly, dis-assembly, rework, test, training, fault isolation, maintenance, installation, calibration, removal, packing, unpacking. The engineering knowledge may cover the complete product engineering lifecycle of all products manufactured by the organization.

[0013] Fig. 1 is a schematic flow diagram of a computerized process 100 for assembly procedure generation of a project portfolio.

[0014] At a first step 102, external content, i.e., input to be stored in a digital portfolio is received by the system and may include the following types of data, as well as additional types not listed:

• Instruments repository

• BOM, EBOM, MBOM

• CAD in various representation formats model and model images

• DOC, DOCX, PDF documents

• JPG, JPEG, PNG, TIFF, GIF, SVG images MP3 videos

• DWG, DXF drawings

• Procedures and sub-assemblies

• Log files, database formats, spreadsheets, data sets

• Manual Engineering Change Orders (ECOs) and Manufacturing Records Books (MRBs)

[0015] At a step 104, the correctness and completeness of the input is validated by the system to ensure: a. No missing data. In particular, the input includes hierarchical lists of parts and their sub-parts, including hierarchical images of parts and their sub -parts. The system confirms that for any part that is an assembly of sub-parts, all subparts are also defined at least by part numbers and/or names and have accompanying images. b. No BOM upload conflicts, no conflicting catalog numbers or part names, no images used for different parts with different catalog numbers or names, etc. c. No conflicts with previously stored data.

[0016] One aspect of the validation includes validating that the part names from BOM documents match part names in the CAD documents (and other image/3D sources, such as blueprints). Labels and quantities are compared, with a focus on semantically significant names. Parts that appear to be identical are flagged to be merged by operators (i.e., users). Computer vision recognition may be applied for this purpose.

[0017] At a step 106, as data is acquired and validated, the data is applied to the generation of a project structure tree of the portfolio, by identifying sub-assemblies of each part. The structure is built of elements that may be defined as objects of classes. A simplified Unified Modeling Language (UML) of main classes of the portfolio is described below with respect to Fig. 2. The class diagram (UML) of Fig. 2 includes two different sections that are tightly coupled:

1. A Structure Graph 202: a graph representation of the product tree.

2. A Procedure Graph 204: a graph representation of different procedures describing a single engineering process of any part in the Structure Graph.

[0018] In addition to these two main sections, the UML indicates information that describes the project content, such as the instruments needed for each process. The graphs use standard UML notation, with filled arrowheads indicating “composite” relationships, and open arrowheads indicating generalization (“type of’) relationships.

[0019] The data for any project is stored as a “container” referred to as a project 210, one or more of which make up a portfolio. Projects can manage one or more products in one installation. Each project encapsulates all engineering production portfolio knowledge regarding a specific product or sub-system, including its BOM and product BOM relations, and all engineering procedures.

[0020] Each project can be nested in a container of a larger project. That is, a project may be a part of a larger project, receiving its own part number. Each project has its own setup containing the configuration, guidelines, and list of instruments used. A setup from one project may be copied to another with local editing to indicate differences.

[0021] A project always has a root item 212, which is the highest level of the structure graph. The root item when assembled is the whole product. A project may also contain additional aspects such as manpower, financials, timeline etc. The project is built from many input sources that are validated as described above. In some cases, certain updated files may have problems with inconsistencies that are flagged, and in some cases automatically corrected. [0022] The project structure (“product tree”) may include multiple instances of reused parts, as needed. Thus, the portfolio representation of the BOM may be implemented as a set of linked tables depicting a graph structure needed for the real representation of part relations in an industrial facility. The creation of inheritance relations between each part and its descendants, also referred to herein as “sub-parts” or “sub-assemblies,” depends on the input that is provided, which triggers diverse methods for building the hierarchy of the structure graph, such as:

• Automatically created hierarchy, based on a spreadsheet list of relations.

• Automatically, by extracting the relations appearing in procedure instruction documents.

• Automatically, by extracting part relations from existing BOM images using vision recognition.

• Automatically, by extracting part relations from a 3D model.

• Manually, through a user interface.

[0023] While building the project hierarchical structure of part relations, a user can reference part records, also referred to herein as Illustrated Part Catalog Items (IPCIs), which may be used in other projects or in the current project in other places. It is possible to create a new IPCI that is generated by assembling other IPCIs together, thus each IPCI can be an assembly of a group of other IPCIs.

[0024] A key task for generating a correct and validated production portfolio is the identification of an assembly’s sub-assemblies, also referred to herein as sub-parts.

[0025] The Structure Graph is defined by the following classes, in addition to the root item: the Illustrated Part Catalog Item (IPCI) 220, the Part 222, the Sub-Part Instance 224, the Assembly 226. [0026] The IPCI class 220 is indicated as having types that may be a part, an instrument, and a process part, and can also comprise other types such as a material or a dedicated aid that may be called a jig. Each IPCI has a unique ID in its project scope. Each IPCI can also be defined as having the following features:

• “Disposable”: the part is to be disposed, if it is disassembled from a parent assembly. This enables validation in rework procedures, to make sure that the IPCI is not used again and must be replaced.

• “Needs calibration”: the part requires a calibration procedure to be valid.

[0027] A part, as indicated (part 220), can be an assembly type, made of sub-parts (child parts), or can be a basic type that has no sub-parts, that is, an item that cannot be disassembled into components. Any part can be referenced multiple times in an assembly and/or in multiple assemblies, therefore the same part data may be reused in the Structure Graph.

[0028] A part can also be defined as any of the following:

• “Attaching part”: a special part that is used to connect other parts together. Examples: screws, washers, nuts, pins, and zip-ties.

• “Alternative part”: an IPCI part that can be used instead of one of the original BOM parts of a product.

• “Accessory part”: an IPCI part that can be added to the final product as an option.

• “MRB part”: a part that is sent to the Material Review Board (MRB committee), to decide which faulty IPCI parts/sub-assemblies should be fixed and reused, or disposed. Usually, it is an expensive part/assembly and the decision is made by an MRB committee.

• “Needs S/N”: a part that requires a serial number to uniquely identify a specific part and for maintenance of the product. [0029] A part of type assembly 226 should indicate two or more assembled sub-part instances 224, which can be of the same part or different parts. That is, the assembly class links two or more sub-parts to a parent part. The “two or more” restriction enables validating that each part indicated as an assembly must have more than one child and should also have an assembly procedure (i.e., an assembly step, as described below) that describes how it is assembled. Each assembly that is part of a higher assembly is a sub-part (also referred to herein as a “subassembly”). In object-oriented terminology, an instance of an assembly class links (or “points”) to instances of part-instance classes. Hereinbelow, for greater clarity, an instance of a sub-part instance is also referred to as a sub-part instance record, such that each instance of an assembly, or assembly record, is said to refer to multiple sub-part instance records, each of which, in turn, is itself an instance of a part class.

[0030] The Structure Graph is built from such parent-child connections between all assemblies and their subassemblies and basic parts. The Structure Graph is referred to as a graph, because many parts are reused between the different assemblies.

[0031] Fig. 3 shows examples of parent-child relations for two interrelated projects, project 310 and project 320. The two projects have unique root items, respectively roots 312 and 322. However, both projects use two of the same types of assemblies, assemblies 330 and 350. The two projects differ in that root 312 also uses an assembly 360, while root 322 also uses a basic part 370. assembly 330 in turn is assembled from assembly 332, and from basic parts 334 and 336.

[0032] The product structure tree reflects the hierarchy of the IPCI assemblies and subassemblies. Each sub-assembly includes all its child IPCI parts or other subassemblies. The product structure tree is really a structure graph tree, as the same parts may be used in different subassembly branches. As described above, the root item is the assembly that integrates all the subassemblies into the final product, that is, the root is the high-level assembly of the product. Each project has only one root item.

[0033] At the end of the project documentation process, validation is performed to confirm that all part quantities declared in the original BOM equal the number of sub-part instances linked to all assembly records (i.e., to all assembly “instances” in object-oriented terminology) and also equals the number of parts referenced in all assembly procedure steps as described below.

[0034] A process part 228 is a part that is a result of an action step. Typically, such a part is an assembly, but may also be a result of a disassembly action step, or the result of processing a basic part. As opposed to the data of the structure graph, which are generated during the data acquisition process, the process part data are typically generated as steps are defined, and therefore can be considered part of the procedure graph. Being generated during step generation, a process part typically does not have a customer catalog number. However, it must have an internal catalog number, which may be set automatically or by the user. Often, a process part is an interim assembly part in a flow of steps (FOS) of assembly (used in an assembly step, that is, “action step,” as described further below). For example, when connecting two parts in an assembly procedure step, the outcome is a new process part that is the compound part generated from this action.

[0035] When connecting a third part to the previous process part, a new process part is generated. Similarly, when disassembling a sub-part from an assembly, the remaining assembly from which the sub-part is detached is also a process part.

[0036] A structure graph tree can be built manually by setting an IPCI part as an IPCI assembly in three use-cases:

1) Root IPCI is an assembly.

2) During BOM import, parts are tagged as assemblies. 3) Any time from the interactive interface, a menu allows conversion of a basic part to an assembly. This is useful if a user did not specify that a part was an assembly during BOM import.

[0037] An IPCI assembly is defined as an IPCI part in two use-cases:

1) During BOM Import, IPCIs not tagged as assembly are defined as parts; and

2) at any time, a menu of the interactive interface allows a document to turn an assembly part into a basic part, if the part does not have IPCI children. Further validations are performed and the system may warn a user (e.g., a project manager) at later stages if an assembly is not on an IPCI Structure Tree (S-Tree) or if no sub-parts are allocated to an assembly. In most cases, this is an error, though a part may have a procedure involving processing of the part without combining sub-parts, but changing its state. For example, a procedure may be applied to a straight rod to turn it into a threaded rod, without the combination of sub-assemblies.

[0038] If no FOS exists for an assembly, this is an error and means that either the assembly should be a standard part (“basic part”), or a FOS must be added.

[0039] Defining a part also includes defining features that affect the generation of engineering procedures. An example of these features is “disposable,” a flag that a part must be thrown away in case of a disassembly, rework, maintenance, or repair. IPCIs can be disposable in case they are to be replaced. An IPCI that has irreversible effects on that IPCI such as spreading LOCTITE to torques to filling a hole, marks the IPCI as a disposable candidate, parts, as well as tools, materials and jigs can be disposable. The relevant instances of the parts must be marked as disposable.

[0040] When images are automatically uploaded from CAD and BOM files, the automation process also flags disposable candidates, as based on a previously defined list.

Documenters with the relevant permissions can manually flag or unflag a part as disposable. Similarly, the “needs calibration” flag may be tagged automatically for both parts and instruments.

[0041] Every disposable candidate that is unflagged is highlighted in a user interface and a project manager may get a warning list of all unflagged disposable candidates.

[0042] For validating the exact quantity of parts usage throughout the assembly procedure, the process manages instances of each part in the project, meaning that usage of each part instance in the documentation is tracked, enabling validation of completeness of the assembly procedures. At a step 108, part instances are managed, each part instance being assigned a Universally Unique Identifier (UUID) in addition to pointing to the part type, by which general features are reused in defining each part. A part instance is defined as each one of the different references of each part in the project. Storing this data enables validating that:

1) There are no “forgotten” instances.

2) The used quantity is exactly as defined.

3) Assembly procedure documentation is complete and can be terminated.

4) Specific part instances that are glued/soldered are disposed of during rework or repair (unless declared otherwise).

[0043] In short, validation of part instance quantities includes: 1) counting each instance of each part in all BOM images, and 2) verifying that each sub-part instance is linked to an assembly, and that it is used in at least one step, meaning no sub-parts have been left out. Hereinbelow “part instances” are also referred to as “sub-part instances,” because each part instance is a sub-part of an assembly.

[0044] The above steps involve acquisition of the raw data including models, streams or still images, texts, spreadsheets, and other information, validated and transformed by store procedures into an internal unified part structure. Engineering rules for manipulating the engineering knowledge are then applied to generate verifiable engineering processes, beginning at a step 110 of Fig. 1.

[0045] At step 110, images are created or acquired to support procedure steps. New images may be created if part images are missing or are of poor quality, or if needed for a Parts Inspection process.

[0046] Images may be acquired from CAD model, or manually added by a user, for later by applying viewpoints generated by a machine learning model trained on the Mechanical Component Benchmark (MCB), a database of 3D models that has been tested in classification tasks by different network models. The extensive size of about 40,000 models in 25 generic categories enables training visual recognition systems with a high level of accuracy and generalization. The system may compute accuracy scores and confusion matrices for each image of, for example, 26 viewpoints, and may select the highest. Assembly procedure BOM images are then selected and part Index Pointers (PIPs) are added to each image for each subassembly (“sub-part”). Whenever a documenter adds a PIP to an assembly procedure, the documenter should indicate whether it is a new instance or one of the previous ones. A PIP is a GUI element having an index number, which is generated for each flow of steps (FOS), which is linked to each assembly. (An assembly procedure FOS may use assembly procedure index-numbering.) Each PIP has a number near it. If the quantity is one, then it is selected automatically. Each image can also have an additional interactive layer with balloons showing the different sub-parts.

[0047] A PIP GUI element includes:

■ a circle containing the index number.

■ An arrow pointing at an IPCI location in the attached image.

■ An IPCI catalog number by the arrow

A list of instance indexes below the arrow [0048] In step display, a PIP may include a black circle and index number. In BOM images, the PIP may include a red circle and index number.

[0049] Balloon indices are also linked to each IPCI and its corresponding illustrations. The process may include:

• Manual marking of balloons on existing uploaded images.

• Manual selection of the point-of-view images created with a smart engine.

• Automatic index generation using vision recognition of balloons/numbers on existing uploaded images.

[0050] Fig. 5 shows an example of presentation of a step image 500, an exploded part image 501 in a screen of an interactive interface for displaying a step instruction, with the addition of PIP balloons 502, including PIP numbering that is also associated with action steps and assembly instruments related to given sub-parts, in a legend 504. The exploded part image (the “main image”) includes two screws, indicated as PIP numbers 6, and two additional elements that are assembled together. The legend of action steps (e.g., “screw,” as indicated) and assembly instruments (e.g., screwdriver) may be automatically added to the presentation of the step image, based on the actions and instruments linked to a given step to which the step image is linked. The main image, that is, the exploded part image 500, is typically derived from an exploded BOM image of the assembly part being assembled by the linked FOS.

[0051] Step images can be created automatically from a model, building the step images bottom up or by exploding the model, saving each disassembly image step and focusing only on the parts involved in that assembly step parts in a reversed order. Alternatively, step images can be collected in a prototype assembly line that documents a step with video or stills. The rest of the step elements are collected automatically and may be generated by an assembly Procedure Generator. [0052] A drawing engine may convert static uploaded drawings to interactive “live” drawings by identifying relevant elements in the drawing. The interactive drawing may be extracted from a model, picture or even a video frame. The source drawings can be, for example, DWG or DXF files, images, or 3D models.

[0053] In electric drawings, elements include connectors and wires. Clicking on a wire highlights it. Clicking on a connector jumps to the same connector in a different drawing and/or zooms in.

[0054] An interactive drawing is created in any of the following ways:

• Manually by marking all elements.

• Automatically processing of an uploaded 3D model such as but not limited to OBJ, STL, DWG / DXF file and automatically identifying all its elements by analyzing its components.

• Automatically processing of an uploaded 3D Model or image file and automatically identifying all its elements by using vision recognition.

• Uploading a drawing file and leaving it static with no interaction.

[0055] Input includes image drawing files, 3D models, and/or elements marked and added manually by user-uploaded files. Output may include a complete interactive drawing with all its components, stored in the database.

[0056] Referring back to process 100 (Fig. 1), at a step 112 the procedure graph 204 described above (with respect to Fig. 2) is created. The IPCI is the main object to which procedures, including assembly procedures, are linked. The IPCI includes workflows, referred to as Flows of Steps (FOSs) that are to be executed step-by-step, with each step requiring verification by the system that the step is completed and correct before a subsequent step can be executed. The workflows are created by the system as linked lists of stepwise procedures. Engineering knowledge generated by the system includes catalog product relations (known as product trees), interactive lists of materials, lists of tools, assembly and other procedures, troubleshooting leading to rework, training materials, specialized drawings, models, animations, etc. Some of the modules of the methods may receive external data sources as input, such as CAD drawings, data and reference and vendor documents. Other modules receive as input the output from other modules. The generated procedures serve diverse users in various ways. For example, engineers may define procedures related to an IPCI to allow manufacturing workers to access training materials through the system as well as to receive step-by-step assembly procedures while working in a production facility. Engineers may view and correct manufacturing digital procedures and create updated versions of the digital procedures and documentation. Managers may generate pdf printed documents for review and compliance purposes.

[0057] In industrial production, each step succeeds if the expected result is reached or fail if the expected result is not achieved. When documenting the production process correctly, it is important to describe what must be done when each step fails. Every step that fails has an alternative path that describes the next steps that must be executed. Each project also has a default alternative FOS, that as default is linked to each alternative path.

[0058] The class diagram encompasses all the validations and processes performed. Every document that is automatically generated by the system is fully compatible with the instruction record.

[0059] Each step of a FOS is defined once and can be reused in different places. Main path flow controls the correct flow between steps. That is, each FOS is a container of one or more steps that can be connected in different paths (i.e., sequences of steps) depending on input of the user who executes the FOS. The FOS is a logical unit that describes a complete process to be executed, therefore it has a meaningful label to describe it. When one or more steps are reused, they must be encapsulated in a FOS container that is identified by a logic label and has an end step for each path. All the FOS paths are part of a graph structure indicating the correct step sequence to be followed.

[0060] A FOS can contain any number of each of the different types of steps. To validate the proper state of a FOS (i.e., no main-path loops), every time it links to another FOS using a link step (also referred to herein as an “L-Step), the system checks whether the FOS links to itself.

[0061] Each FOS always has a scope. The default scope is the IPCI it was declared in (i.e., to which it is linked). FOSs are never nested as a parent-child connection but can reference each other by using a link step, meaning that the referenced FOS must be executed before continuing to the next FOS’s step. A FOS can be referenced by multiple procedures of its own IPCI or by other IPCIs’ procedures, in which case the FOS scope must be extended to a higher IPCI or become global. A FOS becomes a global candidate first, and only after approval (e.g., by a document manager/chief engineer) does it becomes a global FOS for reuse.

[0062] A global FOS cannot be edited but can be duplicated as a new global FOS containing the required changes and is then available for use as a new FOS.

[0063] When a FOS that includes references to specific IPCIs is marked as a global building block, each referenced IPCI number is replaced by a temporary internal catalog number, so that it can be replaced with a real catalog number when used anywhere. In this case, the local duplication is not connected to the original global FOS.

[0064] The scope of an approved global FOS does not belong to an IPCI and is not deleted if the IPCI is deleted. A global FOS can be referenced in any procedure/FOS of any project. A global FOS is visually identified as global in the system’s interactive interface (“GUI”). The FOS container has a link to the global FOS instead of having a local duplication of it.

[0065] Each global FOS stores a reference count of usages (only references in the current Gen installation), to show managers how often it is used. This can be implemented by using the DB count function.

[0066] The core class UML is shown in Fig. 2, showing interaction of steps, FOS and procedures entities and relations. It provides for internal manipulations that relate various content types steps in different main paths that combine various FOSS to create diverse types of engineering procedures. There are two classes of procedures that define performance of specific tasks. Internal Procedures are a series of FOSs that apply to a specific IPCI and all its descendant IPCIs. They include assembly, disassembly, rework, test, fault isolation symptoms and maintenance. External procedures are a series of FOSs that apply to any IPCI. They relate to the IPCI as a black box and include safety instructions, calibration setups, installation, removal, replace, packing, unpacking, and manual procedures. The most common FOS types are indicated in Fig. 2 as: General Procedures 230, Parts Inspection 260, Guidelines 262, Safety Instructions 264, Instrument Inspection 266, and Instrument Tutorial 268. Additional types of procedures are also described below.

[0067] The general, internal FOS procedure is a flow-of-steps (FOS) that contains information and structural rules, allowing it to uniquely describe an engineering process related to a specific IPCI. Each procedure type has its own restrictions, including specification for validating that it is defined well. General rules for setting procedures are that: 1) every procedure must first link to guidelines, safety instructions, and an instrument instruction procedure. An assembly procedure is a special type of procedure that must also link to a Parts Inspection procedure. The parts inspection procedure links to the BOM images 252, described above with respect to step 110 of process 100. Each BOM image includes assembly CAD exploded images or parts photos, Part Index Pointers (PIPs), part quantity and a catalog number.

[0068] The assembly procedure defines steps for creating assemblies from constituent parts, as well as implementing manufacturing processes on individual parts or assemblies. To create an assembly procedure, a Parts Inspection FOS should first be linked to the IPCI. The Parts Inspection FOS includes links to the relevant BOM images enhanced with PIP data, including “exploded” images showing the components (e.g., “parts” and “sub-parts”) as described above. Next an Instrument Inspection FOS and a Materials Inspection FOS should be added.

[0069] The disassembly procedure may be a reversed list of steps of the assembly list of steps, replacing each action with its opposite action, thus it includes parts and instruments inspection, informative, action with input values and links. In the disassembly reversed step list, for each step which has a result, the new step should be vice versa. That is, the result image should be the step image and the step image should be the result image. For actions that have no opposite (like “Perform”) or when the opposite action is not clear, the user may get recommendations of closely related actions and may fix the action manually. The system may add the closely related actions and recommend them to the user in the future.

[0070] A Rework Procedure defines a flow of steps that link to specific steps or FOSs in assembly/dis-assembly procedures for fixing or replacement of parts using informative and action steps with control buttons. Rework may be triggered by a change in an IPCI with all its relevant procedures or by a failure in a FOS test or result.

[0071] A Maintenance Procedure defines a flow of FOSs and steps that may also have links to specific FOSs in assembly/dis-assembly procedures. This procedure allows to choose parts and instruments inspection, informative and link, while actions are typically designed with control buttons. Maintenance aims at guiding the user through the expected procedure via text, image or video. This procedure may define a frequency (schedule) of how often the procedure should be run in production and/or through the product lifecycle. Maintenance can be applied according to three possible events:

• A time event: Mandatory scheduling, setting a frequency (schedule) defined by an engineer regarding when the maintenance procedure should be applied. It fits periodic maintenance.

• A utility function event (Optional). The utility function can be defined by the possible damage estimated by the cost of the possibly damaged parts + waiting time for replacing parts + cost of replacing parts + cost of work + the idle time for repairs + an estimated factor of additional damages (safety or others) divided by the cost of replacement parts + cost of work + plus idle time for repairs. A threshold is calculated and if the function value is over the defined threshold the maintenance procedure is recommended. For a data collection task, the function can be determined by a machine learning (ML) algorithm.

• A broken IPCI event: this procedure would be the same as a rework.

[0072] A Test Procedure defines a flow of informative and action steps, or FOS, to perform a test. Input values can be yes or no, or other values as described in the action step type. It can be formatted as a checklist of individual tests or as in an acceptance test. A Test can also be an automatic test plan (ATP) while interconnecting, that is, getting input from sensors or test equipment or instruments at a test station. More complex tests may also have link steps, parts, and instruments inspection steps.

[0073] A Fault Isolation Procedure may be generated by an automated routine which may create a step-by-step test map based on a set of iterative, informative and action steps with user input to identify a fault, which may then be followed by a Rework Procedure for correcting the fault. A fault isolation symptom identifies a possible fault and the procedure to be performed. Thereafter the fault isolation symptom is assessed according to the highest frequency of likely faults. If symptoms remain, the next highest frequency fault isolation symptom is checked until the fault isolation symptom is removed.

[0074] Each fault isolation symptom can be described by specific text, image, video, sensor, or test value. Each fault isolation symptom is declared under an IPCI and relates to it. Each IPCI may have many fault isolation symptoms. The action steps lead a user through a diagnostic process to one of the (action) result steps to be checked manually by the user or automatically by machine vision or another automated algorithm. Each result step is assessed and linked to the rework procedure to remove the symptom. This process may be repeated until the symptom is removed. At the end of the procedure a conclusion will be conveyed to the user in the form of informative steps that will include:

• A message to the user.

• The symptom reason that caused the symptom to happen.

• The rework procedure/s that fixed it

• The indication can be performed via text image or video if requested so by the user.

[0075] A conclusion in a fault isolation procedure can be several informative steps followed by action step with a link as a value. Different symptom routes and steps are created in one of the following ways:

• Manually by using the smart symptom step engine.

• Automatically processing of an uploaded symptom file (e.g., DOC, DOCX, CSV, or PDF file) and automatically creating the full symptom step-by-step routes map, input controls, rework and the conclusion steps.

Manually marking steps in a file for cropping. Using vision recognition to mark steps in a file for cropping

[0076] Input to fault isolation may include original images/videos, text, sensor or test value and all test/conclusion steps and other content added manually by a user as uploaded files. Output may include a complete fault isolation symptom relations set with all its test steps and conclusion steps in the database.

[0077] An Install Procedure defines a FOS that describes how the specific IPCI must be installed in its father IPCI. It includes parts and instruments inspection, informative, action and link steps. In every instance where this IPCI is installed, there must be a link step to the install procedure.

[0078] A Removal Procedure defines a FOS that describes how the specific IPCI must be removed in every place it is used. It includes parts and instruments inspection, informative, action and link steps.

[0079] Replacement Procedures are automatically generated procedures that link to removal and then install procedures of a specific IPCI. The last FOS in the replace procedure is an action with a user input check or a machine or sensor input for correctness verification. [0080] A Packing Procedure defines a FOS that describes how the specific IPCI must be packed. Packages are kind of jigs that allow single and assembly IPCIs to fit exactly into that packing molder to avoid free moving during transport. It includes safety procedures, parts and instruments inspection, informative and action steps with links input.

[0081] An Unpacking Procedure defines a FOS that defines how a specific IPCI must be unpacked. It includes all the safety procedures, warnings and preparations needed before the unpacking of the IPCI to allow safe delivery. It includes parts and instruments inspection, informative, actions steps with links and tests results inputs.

[0082] General Guideline (“GG”) Procedures include a list of special actions that need to be executed prior to execution of any procedure. This list should be added automatically at the beginning of each sequence of procedures. Each procedure type, such as assembly and disassembly, have a specific list of guidelines. Safety Instruction (“SI”) Procedures are a specific type of guidelines that are also typically added, including informative and action steps that must be performed before starting a procedure. It is possible to add mandatory event driven triggers such as when a procedure has stopped more than 40 minutes then run a safety procedure again before any action that might endanger the worker. The Safety Instruction and General Guideline procedures are also often grouped together and referred to herein as SIGG Procedures.

[0083] Parts Inspection procedures include action steps which help the user to check that all IPCIs to be used in a procedure exist. This FOS also includes the BOM image, which shows a visual display of some or all assembly parts with their corresponding IPCI numbers and balloons for ease of part detection and verification.

[0084] Instruments Inspection procedures include inspection action steps for the user to check that all needed instruments to be used in a procedure exist.

[0085] Tutorial procedures include a defined flow of steps to use an instrument, such as a tool or jig, typically in conjunction with a part. It is typically an event driven procedure that is mandatory at least for a first-time user of a part/tool/jig. Calibration setup of a part/tool/jig is also a kind of tutorial that is used for resetting it to the manufacturer definitions. Training procedures are of a of a particular type because they can be created for every procedure of any type, thus a training engine, like the jig training engine 110, generates interactive training material, assigns a Procedure ID for step-by-step training, and requires defining the criteria for users to pass the training. There are several types of training:

• Tutorial - A procedure that provides an overview of the topic, then zooms in - a demo of the topic, and last details on demand (details on specific functions). Tutorials can be mandatory or optional defined at the project configuration or in the procedure. They are offered the first time a user performs a new task.

• Certification - A tutorial with an evaluation task that is required for performing specific procedures. It is mandatory.

• Competency - Demo only for the purpose of periodic refreshing of specific topics. It is usually mandatory, requesting the project manager to define its frequency (schedule). It can be changed to optional at the project configuration or in the procedure.

[0086] A step (242), shown in the class diagram 200 of Fig. 2, is the basic process element (or “record”) for describing a single executable phase of a flow of steps. That is, steps are the building-blocks of procedures and their FOSs. A step includes:

1) Step Content 244: typically the BOM content, which may include before and after images, edited with PIPs, as described above.

2) Condition 246: the condition that is tested after execution of a step to check if the real outcome is identical to the expected result. The condition can be defined as a simple yes/no condition or as a formula to be tested with respect to the result.

3) Result: the expected result of performing a step, with respect to a defined condition. The result is typically indicated by a Boolean indicator, which is a simple yes/no flag, or a number, a numerical range, or an image compared that is then compared with a visual result.

4) Step Flow: according to the condition and the result, a pointer to a next step for a valid result, and a pointer to an alternative step if the flow stopped because of an improper state or invalid result. The step flow can also be a link 250 to a new

FOS. [0087] There are several types of steps, including: action steps, informative steps, check steps, and alternative steps. The types of steps are determined by their content.

[0088] Action steps are the most common step type and describe an action to be done in the procedure. An action step should have an action step showing the result of the previous step in which the user verifies that the outcome of his action is correct. Action steps may be automatically generated after collecting the following elements:

1. A textual instruction (Step name)

2. At least one main image

3. Action icons, which are icons selected from a predefined list

4. At least 1 PIP indicating the parts in the image.

[0089] An Action Step may also include a result and condition, as described above, as well as list of relevant instruments for performing the step, and additional remarks.

[0090] Actions in a step can indicate the relevant instances of the parts that must be marked as disposable such as: Action Torque, Action Weld, Action Ream, Action Solder.

[0091] Informative steps are steps without an action but used to provide a user with important information. It can be a step with text for adding instructions that can be declared by bulleted or unbulleted paragraphs. It can be automatically created by merging several sentence steps into a bulleted informative step. Informative steps contain text, bullets, numbering, images, tables, video, or model, and/or paragraphs of general information relevant for processing the procedure correctly. The ability to add informative steps can be used in any type of procedure. It is used when the user wants to add a step that should only show information for the user without any action to be taken. [0092] A Check step requires a user input to be processed, and the system refers to the next step/FOS accordingly. The decision is taken according to a condition/test.

[0093] A String step type requires a string user input before moving on to the next step (e.g., “Enter the machine error number shown,” or “Enter a QA provider code”).

[0094] A Barcode step type requires a user to scan a barcode and is used either in a Part Inspection FOS when an IPCI requires a serial number (“S/N”) input, or in a Rework or Maintenance procedure when a faulty part is removed, and its S/N should be scanned.

[0095] A Link step (“L-Step”) type is a link to a complete FOS or procedure, allowing the reuse of a series of existing steps, and then getting back to the main flow of an FOS. It aims to reduce needed fixes in duplicated content.

[0096] An End step type allows a Documenter to confirm that a FOS is considered complete (both main-path and alternative-paths) and triggers an execution of procedure validations and alert the Documenter if a FOS is not completed properly.

[0097] Creating steps requires taking one or more of the following actions:

1. Selecting an image of an IPCI to act upon.

2. Selecting the IPCIs appearing in the image that are relevant to the current step from the procedure IPCI table. The selected IPCIs are connected to the catalog in the database. The IPCIs are represented by balloons appearing in the assembly image. Arrows connecting the balloons with the location of the IPCI in the image are added automatically and can be replaced manually.

3. Adding highlights, which may be, for example circles or rectangles that indicate important areas of the step image. They aim to get the user’s attention. Additionally, it is possible to add a zoom-in image to a selected highlighted area.

The zoom-in image is automatically generated and can be replaced manually or automatically using a predefined directory and a predefined number of the image to be replaced afterwards by an image edited in the image tool. When there is more than one option a popup selection window should appear afterwards by an image edited in the image tool.

4. Selecting the relevant engineering actions from the actions glossary list to be performed on the parts. Actions are semantically constrained. Each action must be either connected to a part catalog number selected from the step part list or to a jig. An Action Category spreadsheet would be an example of the action categories table. The table gets additional action categories and actions as we define new projects and disciplines.

[0098] Action steps can also include the following:

1) Tool - iconic representation/text of the tool to be used to perform the action.

2) Materials: iconic representation/text of the material to be used to perform the action.

3) Jig: iconic representation/text of the jig (special dedicated tool designed to help perform special actions like press, align and others to be used while performing the action).

4) Textual comments - A short additional description relating to the action.

5) Adding Remarks -Add free text anywhere in the step to clarify and comment.

6) Adding Videos -Add short videos to help clarify complicated steps for the purpose of step demo, training, and tutorials.

7) Adding Models: Add 3D models, illustrations such as composer or similar illustration and animations to allow live direct manipulation of the assembly parts participating in the step or allow view generation including recommending the best view for selection of the step image. 8) Adding Links - It is possible to add a reference link to a different step in the same procedure for enabling repetitions.

[0099] Action steps also link to process parts 228. Usually, a process part is an assembly that is the outcome of a previous assembly/disassembly action step, as can be seen in the class diagram (the connection between step content type and the process part). Process parts may also link to other process parts, reflecting a combination of parts in an assembly.

[00100] Fig. 4 shows an example of steps of a Procedure Process 400 linking to process parts of a process part Tree 402, based on a Part Structure Tree 404. The Part Structure Tree is taken from the example of Assembly 330 of Fig. 3. The FOS 410 includes a first step 420 that assembles parts 332 and 334, to create a first process part 450. The step 420 includes an action 422 of comparing the process part result with a desired result, and a condition 424 leading to the next step 430 if the result is valid, and to alternative step 440 if not. At the next step 430, the process part 452 is the result of the step action, combining the process part 450 with the basic part 336. The step 430 compares the process part result with a desired result 330, and a condition 434 leading to the next step, which is the FOS End 442 if the result is valid, and to the alternative step 440 if not.

[00101] Figs. 6-7 show a flow of an assembly procedure with rework, showing the control flow between steps, FOSs, and procedures.

[00102] Fig. 6 shows a simple scenario of an assembly procedure in which nine steps are performed, including steps of a rework procedure, a disassembly, and fault isolation.

[00103] The tables in Fig. 7 show how the same scenario is saved in linked tables, emulating the UML of Fig. 2, where the keys of each table are used in the other tables. Table #5 shows records of the steps of Fig. 6, each numbered with a “step Id” key indicating the step number and a “step content Id” key linking to step content stored as labeled records in Table #1. For example, steps 1 and 2 point to respective step content records 1 and 2, which are “screw” action steps. Steps 3, 5, and 7 have conditions, meaning that the step flow is dependent on the result outcome. As indicated, when the result of step 3 is not within the target tolerance specified by the step condition, the “alternative” step that follows (for an invalid result) is step 4, which is a link to a disassembly FOS, which is indicated as a record of Table #3.

[00104] If the disassembly result is invalid, the next step (step 6) is a link to a fault isolation FOS. The steps continue until either a valid or invalid step result leads to an end step, as described above.

[00105] Table #4, the “IPCI FOS Table,” indicates the FOSs that link to the assembly ABC. In Table #7, the sub-parts of ABC are listed (as well as the fact that ABC is itself a sub-part of a higher level assembly). Note that all sub-parts and process parts are also part records in the IPCI table, Table #8. Also, note that ABC includes two screws, which are given separate indices in the part instances table, Table #6. Finally, note that each step record of the step content table, Table #1, also refers to the relevant part instance used in the step (that is, a reference to a part instance record in Table #6) and to the process part generated by the step (a reference to a part record in the IPCI table, Table #8).

[00106] Referring back to Fig. 1, once the Procedure Graph and Structure Graph are complete, the instruments are catalogued at a step 114 by an instrument management module, which performs tasks such as differentiating between global instruments and project local ones. Instruments include materials, tools and specific or generic jigs used to help describe the procedure step actions in the different procedure instructions of the projects. Jigs are special tools that are designed to help with a specific assembly /disassembly /rework including one or more actions, usually to be used and executed with a specific project.

[00107] Each instrument may have several types of visual representations associated with the instrument, such as an image, an icon, a name, an ID, etc. The different visual signs can be used either: for specific actions that are filtered automatically and allowed for user selection in an (illustrated) list or manually for every action in any procedure instruction step as per a user selection.

[00108] All the different instruments are categorized to fit the different actions used in the different procedure instructions. For example, a hammer tool and a glue material cannot be used together with a screw action, which can however be used with any kind of screwdriver.

[00109] Instruments that are defined as global are recognized in all projects and can be used to help describe the procedure step actions. The local instruments are recognized only inside the project for which they are defined.

[00110] When a user changes a local instrument to be global, it is automatically recognized in all projects, However, if a user changes a global instrument to a local one, it will create a duplication of the instrument with a local attribute and a new reference ID.

[00111] When changing local instrument parameters, the parameters of the same instrument in other projects are not affected. However, when changing a global instrument’s parameters, all usages of the instrument, except for instances in a locked version, are changed. Input to the instrument manager includes instrument images and data added or edited manually by a user. Output is a list of instruments ready for use in projects, depending on global and local access defined for each one of them.

[00112] Prior to “freezing” a portfolio for manufacturing, all instruments used in steps of all procedures are copied to an inspection table to serve for validation of usage.

[00113] Referring back to Fig. 1, once the Procedure Graph and Structure Graph are complete and the instruments are catalogued, the portfolio is validated at a step 116 before being “frozen” to create on-line interactive documentation for manufacturing and for the other target procedures. The generation and validation of a manufacturing file is a recursive process that is done per each assembly at the low level, then checking that the parent has additional child assemblies to be generated, completed and validated before the generation of its parent assembly. This iterative process is repeated for each assembly till the main IPCI (the final product). This recursive generation with all its components validation (structure tree, procedure tree and process part tree), ensures completeness of the portfolio.

[00114] Validation steps include validating that all PIP-numbered IPCIs are mentioned in: a. The procedure steps at least as many times as declared in the BOM table. b. All BOM images equal the number declared in the BOM table. c. No IPCI is forgotten.

[00115] Any IPCI (part/Instrument) may have a SIGG procedure that is relevant for all links to the IPC. When such an IPCI with a SIGG procedure is generated, an assembly generator algorithm may prompt of its children-parts too and add at least one L-Step to each one of them. A final validation process may then be run going over all “main path” steps and all alternative paths of each FOS, checking that all assemblies have been closed. Warnings of incomplete procedures and assemblies are provided to operators.

[00116] At a step 118, a project is “frozen” for manufacturing, by which all procedure FOSs and steps, used tools, materials, jigs are saved in a repository, for example by storing in a JSON repository, with additional versioning metadata. Versioning also permits a restore process by which a current version is replaced by a prior version.

[00117] Input for the freezing process may include any procedure completed in the editing database. Freeze and restore procedures create a snapshot of the portfolio, which may be performed at the atomic level of procedures and assemblies. Freezing (“versioning”) of all procedures may be performed at a rate set by a user. [00118] The versioning process tracks and manages content changes between different versions of an IPCI and IPCI procedures. A product is a special case of an IPCI at the highest level. A project is a product wrapper, which also includes specifications of management capabilities, such as permissions related to the product portfolio.

[00119] The version creation process of an IPCI is done on its own metadata, such as name, image/s, etc., and on each of its procedures. The IPCI version creation process can be done in many stages, each time with an approval process for new procedures. When starting a version creation process of an assembly IPCI that has descendants IPCIs, the version creation process is recursively done on each of its descendants.

[00120] Prior to signatures the product portfolio with all its procedures is ready for a dry run evaluation. Procedures get comments, updates and revalidation.

[00121] A version creation process is complete, only after it is approved by all relevant approver assignees that are defined by the project manager in the approver assignees list. After a version is created it is locked for editing turning it into a FREEZE status. If one of the approver assignees rejects the version in the version approval process, the relevant procedure/IPCI is unlocked for editing and a new version needs to be created for approval. When a version is fully approved by all approver assignees, the version is totally locked and becomes revised. Opening the procedure/IPCI version for editing or fixing can be done only by its owner (usually the project manager) and a new version creation process will start.

[00122] There are some dependencies between all IPCI metadata and procedure versions, such as disassembly, which depends on the assembly procedure, and rework, which depends on assembly and disassembly. Therefore, an IPCI might have some procedures versioned and approved, while some others might still be in editing. Even after an IPCI is revised and approved (locked for editing) additional procedures can be added to it. [00123] Only procedures that are error free may be candidates for their version creation. In the process of creating a new IPCI version, the user can select from an IPCI procedure list only the procedures that are ready for approval and are error free. By default, all procedures that are error free are selected, enabling a one-click IPCI version creation. Creating a version of an IPCI procedure of an IPCI that has descendants means creating a version of the same procedure type in all its descendants, so they must be error free too.

[00124] To complete any version creation process and become a revised version, each version must be approved by all defined approvers assignees. In case the approvers assignees list is not complete the project manager is notified to complete the list before starting the approval process.

[00125] When a new version is created, notifications and emails are sent both to the project manager to inform him and to the first approver assignee in the list. Every assignee has a defined deadline to complete the approval process. A notification nagging process will remind the assignee to complete the process.

[00126] In the version approval process, an assignee can go through all relevant content of the IPCI procedure and approve every step or reject any step by adding a rejection note to it. The assignee may decide to rely on the previous assignee and skip to the end of the process, clicking an approval button for the whole procedure. In case of at least one rejection, the whole procedure may be unlocked, and notifications and emails may be sent to all previous assignees and the document manager to inform them of the need to fix the procedure.

[00127] In case of a rejected procedure version, a document manager can skip to the rejected steps of the procedure to fix them and create a new version for approval. The approver assignee can skip to the added/edited steps to shorten the approval process. [00128] Versioning of a project also includes several different project sections. Each project section has its own versioning process and counting. Here is a list of different project sections:

1) Project metadata - project parameters and version parameters that enable identifying the version uniquely.

2) Configuration - a set of all parameters and PDF templates defined in the project.

3) Guidelines - guidelines are a list of special important actions which need to be executed prior to execution of any procedure. This list must be added at the beginning of each procedure automatically. Each procedure type has its specific list of guidelines.

4) Instruments - a list of all instruments (tools, materials, and jigs) used in the current project.

5) All IPCI and IPCI Procedure content - the procedure metadata for the relevant IPCI (and all its descendants too), all procedures, all BOM images, all FOSs, and all steps.

6) Global FOSs - a list of all global FOSs which were referenced by the procedure’ s FOSs.

[00129] After a final approval of a portfolio version, it becomes a locked revision, and the following actions are performed:

1) The new approved revision is added to a dedicated database for approved revisions of the project.

2) If the revised procedure is one of the following types: assembly, installation, or packing, then a new disassembly, removal, or unpacking (respectively) procedure may be automatically created. The new procedure may be created by an algorithm that reverses each of an assembly FOS’s steps and applies relevant changes to the step content, while adding suggestions to each step for the documenter.

Following, rework, maintenance, fault isolation, training and other procedures can be prompted for automatic draft creation. A notification and email may be sent to the project manager to inform him that the automatic draft procedure is ready for revision.

[00130] After locking/freezing of a version, the portfolio may be used through an interactive interface at a step 130 to guide interactive assembly, maintenance, repair etc.

[00131] At a parallel step 132, a DOCX/PDF output file may be automatically generated and may be attached to a portfolio for compliance and backup reference. A PDF Engine Generator may automatically create files (e.g., DOCX/PDF files) for compliance and backup reference by merging the relevant data from the database on the fly into a customer or project template layout based on the customer’s template file in the configuration module. [00132] A project manager can define how a file WORD/PDF layout is to be created. The definition contains: revision cover page, content cover page, and the default layout for all content file pages.

[00133] The engine collects all relevant data from the database and layouts all content on the correct pages in the right order, while automatically creating a table of contents (TOC) with the matching page numbers. To prevent accidental editing of the output file, the output file is created as one image per page. This verifies that editing of the content can only be done by the system. The engine may perform OCR analysis, adding the recognized text data into DB tables for editing. An output file can contain:

• A catalog of BOM images and BOM parts table information.

• A procedure with all its steps and FOSs.

• A list of troubleshooting symptoms with all steps and FOS displaying the logic of troubleshooting route maps. Drawings showing all components.

• Header, footer, logos and general and safety instruction for the project

[00134] The portfolio may appear in a PDF containing a. Cover pages. b. TOC pages. c. The Project GG FOS steps. d. The Project SI FOS steps. e. The Project GG FOS steps. f. The Project SI FOS steps. g. Instruments tables: automatically generated and filled when generating a PDF file, including tool, jig, and material tables. h. A Full BOM table. i. Sub-assembly lists including: a sub-assembly BOM table, BOM images from the assembly procedures, and FOS main path assemblies. j. Final assembly data may include: i. Final BOM table. ii. BOM Images from the assembly procedure FOSs. iii. assembly procedure main path and alternative steps.

[00135] At a step 134, the interactive project portfolio may also be presented in an interactive portfolio viewer having a structure similar to the PDF/Word document of step 132, described above. [00136] The system as described herein provides organizations with multiple benefits. These include:

1. Consolidation of documentation by all divisions-R&D, engineering, and production-into one system enabling internal communication between all divisions through shared validated engineering model-based files.

2. Consistency, standardization, and up-to-date digital knowledge base of manufacturing information in a production portfolio.

3. A generative engineering visual language model similar to known Al language models.

4. Automatic generation of validated and verifiable engineering procedural knowledge based on the IPCI, actions, step, FOS and procedures data objects and its internal flow representation.

5. A self-guided automatic and semi-automatic proactive correct, faulty and alternative step flow, allowing interaction with users of all divisions, including for training, reuse, updating, and approval.

6. An automatic generation of triggered execution code from engineering procedural knowledge to allow a full updatable, reusable, revised and validated interactive step-by- step production guide.

7. An automatic generated IPCI graph displayed as product tree, with checks for integrity and correctness of the parts and the procedures, also showing status of completeness of each element and proactively prompting a user for completeness.

8. Recommendations for production line structure, based on procedure/step professions needed.

9. A generative engineering visual language structures the generation of engineering procedural knowledge. An automatic document creator engine collects all relevant data from the database and lays out all content in the right order, for a fully compatible automatic pdf, word, or video document, while automatically creating a table of contents (TOC) with the matching page numbers. The document creator engine generates a full document based on the whole portfolio or based on selective content or update of a specific section of content, by comparing changes made to the database. For optimized performance, the engine may produce only sections of the document that have changes.

[00137] It is readily apparent that the various methods and algorithms described herein may be implemented by appropriately programmed general purpose computers and computing devices. Typically, a processor (e.g., one or more microprocessors) will receive procedures from a memory or like device and execute those procedures, thereby performing one or more processes defined by those procedures. Further, programs that implement such methods and algorithms may be stored and transmitted using a variety of media in several manners. In some embodiments, hard-wired circuitry or custom hardware may be used in place of, or in combination with, software procedures for implementation of the processes of various embodiments. Thus, embodiments are not limited to any specific combination of hardware and software.

[00138] A "processor" means any one or more microprocessors, central processing units (CPUs), computing devices, microcontrollers, digital signal processors, or like devices.

[00139] The term "computer-readable medium" refers to any medium that participates in providing data (e.g., procedures) that may be read by a computer, a processor or a like device. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, solid state memory, optical or magnetic disks and other persistent memory. Transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise a system bus coupled to the processor. Transmission media may include or convey acoustic waves, light waves, and electromagnetic emissions, such as those generated during radio frequency (RF) and infrared (IR) data communications. Various forms of computer readable media may be involved in carrying sequences of procedures to a processor. For example, sequences of procedure (i) may be delivered from RAM to a processor, (ii) may be carried over a wireless transmission medium, and/or (iii) may be formatted according to numerous formats, standards, or protocols, such as Bluetooth, TDMA, CDMA, 3G. [00140] Where databases are described, it is understood by one of ordinary skill in the art that (i) alternative database structures to those described may be readily employed, and (ii) other memory structures besides databases may be readily employed. Any illustrations or descriptions of any sample databases presented herein are illustrative arrangements for stored representations of information. Any number of other arrangements may be employed besides those suggested by tables illustrated in drawings or elsewhere. Similarly, any illustrated entries of the databases represent exemplary information only; one of ordinary skills in the art will understand that the number and content of the entries can be different from those described herein. Further, despite any depiction of the databases as tables, other formats (including relational databases, object-based models and/or distributed databases) could be used to store and manipulate the data types described herein. Likewise, object methods or behaviors of a database can be used to implement various processes, such as the described herein. In addition, the databases may, in a known manner, be stored locally or remotely from a device that accesses data in such a database.

[00141] The present invention can be configured to work in a network environment including a computer that is in communication, via a communications network, with one or more devices. The computer may communicate with the devices directly or indirectly, via a wired or wireless medium or combination of communications means. Each of the devices may comprise computers, adapted to communicate with the computer. Any number and type of machines may be in communication with the computer.

[00142] Human tasks described above by role titles, such as document manager, may be assigned to any operator and given any title.

[00143] EXAMPLES [00144] Exemplary embodiments of the present invention provide methods and systems for generating an interactive production portfolio for manufacturing and/or maintaining a product. Steps of a method provided by a first example of the present invention may include:

1) receiving a plurality of engineering documents and computer-aided design (CAD) models, including a hierarchical list of parts and their sub-parts, and hierarchical images of the parts and their sub-parts;

2) for each part with sub-parts, storing in the production portfolio a part record linked to respective sub-part instance records, wherein each sub-part instance record in turn links to a part record;

3) for each part with sub-parts, storing in the production portfolio an assembly record of the part linked to respective sub-part instance records, wherein each sub-part instance record indicates a use of a given sub-part and wherein all sub-part instance records of a given sub-part in turn link to a common part record with common documentation;

4) linking to each part record in the production portfolio a link to one or more flows of steps (FOSs), wherein a step, for at least one of assembling or disassembling the part from its sub-parts, indicates: 1) one or more sub-parts involved in the given step, indicated by a bill of materials (BOM) image and/or by text, 2) a validation condition for determining a valid or invalid result of the step, and 3) a next step to perform for the valid result and an alternative step to perform for the invalid result;

5) presenting steps for manufacturing or maintenance of a part of the portfolio in a graphical, step-by-step manner by accessing the linked FOS of the part, including the validation condition for each step result of each step, receiving a result, determining conformance with the step’s condition, and presenting the step’s next step when the result is valid and the step’s alternative step when the result is invalid. [00145] An example 2 of the present invention is a method include steps of example 1, where receiving the engineering documents may also include receiving a list of instruments to be used in the manufacture and/or maintenance of the product, wherein the step for at least one of assembling or disassembling the linked part from its sub-parts further indicates at least one of the instruments in the list of instruments, and wherein the method further comprises verifying that each instrument indicated in the instrument list is indicated in at least one step of at least one FOS and that no step indicates an instrument not in the instrument list.

[00146] An example 3 of the present invention is a method including steps of either of the above examples, where the BOM image indicates positions of the one or more sub-parts before and after performance of the step.

[00147] An example 4 of the present invention is a method including steps of any of the above examples, where generating linking balloon indices for each BOM image, including part numbers indicating sub-part identities. In an example 5, generating the balloon indices includes at least one of: receiving, at a user interface, an interface for marking balloons on the BOM image; receiving, at the user interface, one or more point-of-view BOM images, generated by artificial intelligence manipulation of an initial BOM image, for indicating a preferred point-of-view; calculation of an optimal spread of balloon indices to be presented on both sides of the BOM image; and splitting a number of balloon indices between several images according to an order of assembly.

[00148] An example 6 of the present invention is a method including steps of any of the above examples, where the common documentation of each common part record is accessible to processes accessing any of the sub-part instances linked to that common part record.

[00149] An example 7 of the present invention is a method including steps of any of the above examples, where at least one step of at least one FOS is a link to an additional FOS for reuse of standardized FOSs. [00150] An example 8 of the present invention is a method including steps of any of the above examples, where the engineering documentation includes one or more of: text, images, video, 2D models, 3D models, and spreadsheets.

[00151] An example 9 of the present invention is a method including steps of any of the above examples, further including generating the hierarchical list of relations between parts and their sub-parts from one or more of: a spreadsheet list of part relations; part relations identified in procedure instructions; part relations in BOM images, identified by computer vision recognition; and part relations appearing in a 3D CAD model.

[00152] An example 10 of the present invention is a method including steps of any of the above examples, further including representing the interactive production portfolio by a structured engineering procedural visual language enforcing standardization throughout an organization.

[00153] An example 11 of the present invention is a method including steps of any of the above examples, further including presenting an interactive interface guiding a user through the generation of each FOS, including creation of linked FOSs including one or more of a safety FOS and a general guidelines FOS.

[00154] An example 12 of the present invention is a method including steps of any of the above examples, further including presenting an interactive interface guiding a user through the generation of each FOS, including validation conditions that include validation by one or more of machine vision, similarity algorithms, rule based, semantic context based generation, machine, deep learning, data augmentation and simulation, and algorithms.

[00155] An example 13 of the present invention is a method including steps of any of the above examples, further including presenting an interactive interface guiding a user through the generation of each FOS, including a top-down, bottom-up, or singleton creation process. [00156] An example 14 of the present invention is a method including steps of any of the above examples, further including presenting an interactive interface prompting a user to choose an engineering procedure type for a FOS to be generated and guiding the user through a FOS creation process according to the procedure type.

[00157] An example 15 of the present invention is a method including steps of any of the above examples, further including presenting an interactive interface prompting a user to choose conditions for step validation with input having a form of one or more of: a “check” type to select a yes/no response, a color response, a numeric or alphanumeric response, a range of values that can be integer or floating point numbers, a picture to be matched, and a video to be analyzed.

[00158] An example 16 of the present invention is a method including steps of any of the above examples, further including presenting an interactive interface prompting a user to make graphical additions to the BOM image including one or more of additional text, images, models, balloons, and graphical signs.

[00159] An example 17 of the present invention is a method including steps of any of the above examples, further including defining a FOS as a global component accessible by one or more additional interactive production portfolios.

[00160] An example 18 of the present invention is a method including steps of any of the above examples, further including checking completeness of all FOSs by confirming that all parts having sub-parts have one or more associated FOSs and that all FOSs include only parts in the hierarchical list.

[00161] An example 19 of the present invention is a method including steps of any of the above examples, further including presenting an interactive interface prompting a user to define triggering events for FOS execution including a time event and/or a fault event. [00162] An example 20 of the present invention is a method including steps of any of the above examples, further comprising generating a type of FOS as a training procedure to present steps to be learned by a user, that, when presented, provide certification for a specific task or real-time assistance to a user.

[00163] An example 21 of the present invention is a method including steps of any of the above examples, further including presenting an interactive interface providing a graphical measure of a FOS status including at least a status from the following status list: 1) procedure lacking content; 2) detected errors in procedure; 3) validated procedure; 4) approved procedure.

[00164] An example 22 of the present invention is a method including steps of any of the above examples, further including logging to a log file parameters of user interaction with an interactive interface, for generating the interactive production portfolio, including recording user actions, system actions, time stamps, tool usage, jigs, materials, specific combinations of procedures, FOSs and steps, to standardize and improve an engineering knowledge base, and wherein the log file identifies steps and procedures that are incomplete, unvalidated, unapproved, erroneous, and/or conflicting.

[00165] An example 23 of the present invention is a method including steps of any of the above examples, further including providing, by the interactive production portfolio, content reusability of projects, parts, procedures, and steps, updating of content, managing changes, comparing project versions, and propagating changes while requesting pre-defined, necessary user approval for creation of a new project version.

[00166] A further example of the present invention is a system for generating an interactive production portfolio for manufacturing and/or maintaining a product, where the system includes a computer processor having associated non-transient memory with instructions that when executed implement steps of any of the above method examples. [00167] It is to be understood with respect to the above description that although various features of the invention may be described in the context of a single embodiment, the features may also be provided separately or in any suitable combination. Method steps associated with the system and process can be rearranged and/or one or more such steps can be omitted to achieve the same, or similar, results to those described herein. Although the invention may be described herein in the context of separate embodiments for clarity, the invention may also be implemented in a single embodiment. Certain embodiments of the invention may include features from different embodiments disclosed above, and certain embodiments may incorporate elements from other embodiments disclosed above. Meanings of technical and scientific terms used herein are to be commonly understood as by one of ordinary skill in the art to which the invention belongs, unless otherwise defined.

Claims

1. A method of generating an interactive production portfolio for manufacturing and/or maintaining a product comprising:

(i) receiving a plurality of engineering documents and computer-aided design (CAD) models, including a hierarchical list of parts and their sub-parts, and hierarchical images of the parts and their sub-parts;

(ii) for each part with sub-parts, storing in the production portfolio an assembly record of the part linked to respective sub-part instance records, wherein each sub-part instance record indicates a use of a given sub-part and wherein all sub-part instance records of a given sub-part in turn link to a common part record with common documentation;

(iii) linking to each part record in the production portfolio a link to one or more flows of steps (FOSs), wherein a step, for at least one of assembling or disassembling the part from its sub-parts, indicates: 1) one or more sub-parts involved in the given step, indicated by a bill of materials (BOM) image and/or by text, 2) a validation condition for determining a valid or invalid result of the step, and 3) a next step to perform for the valid result and an alternative step to perform for the invalid result;

(iv) for each part record in the production portfolio, verifying that each linked sub-part instance record is also indicated as a sub-part in a step of the linked FOS, and vice- versa, that each sub-part linked to the steps of the linked FOS is also linked as a subpart instance record to the part record;

(v) presenting steps for manufacturing or maintenance of a part of the portfolio in a graphical, step-by-step manner by accessing the linked FOS of the part, including the validation condition for each step result of each step, receiving a result, determining conformance with the step’s condition, and presenting the step’s next step when the result is valid and the step’s alternative step when the result is invalid.

2. The method of claim 1, wherein receiving the engineering documents further comprises receiving a list of instruments to be used in the manufacture and/or maintenance of the product, wherein the step for at least one of assembling or disassembling the linked part from its sub-parts further indicates at least one of the instruments in the list of instruments, and wherein the method further comprises verifying that each instrument indicated in the instrument list is indicated in at least one step of at least one FOS and that no step indicates an instrument not in the instrument list.

3. The method of claim 1, wherein the BOM image indicates positions of one or more subparts before and after performance of the step.

4. The method of claim 1, including generating linking balloon indices for each BOM image, including part numbers indicating sub-part identities.

5. The method of claim 4, wherein generating the balloon indices comprises at least one of:

(i) receiving, at a user interface, an interface for marking balloons on the BOM image;

(ii) receiving, at the user interface, one or more point-of-view BOM images, generated by artificial intelligence manipulation of an initial BOM image, for indicating a preferred point-of-view;

(iii) calculation of an optimal spread of balloon indices to be presented on both sides of the BOM image; and

(iv) splitting a number of balloon indices between several images according to an order of assembly.

6. The method of claim 1, wherein the common documentation of each common part record is accessible to processes accessing any of the sub-part instances linked to that common part record.

7. The method of claim 1, wherein at least one step of at least one FOS is a link to an additional FOS for reuse of standardized FOSs.

8. The method of claim 1, wherein the engineering documentation includes one or more of: text, images, video, 2D models, 3D models, and spreadsheets.

9. The method of claim 1, wherein the hierarchical list of relations between parts and their sub-parts is generated from one or more of:

(i) a spreadsheet list of part relations;

(ii) part relations identified in procedure instructions;

(iii) part relations in BOM images, identified by computer vision recognition; and

(iv) part relations appearing in a 3D CAD model.

10. The method of claim 1, wherein said interactive production portfolio is represented by a structured engineering procedural visual language enforcing standardization throughout an organization.

11. The method of claim 1, further comprising presenting an interactive interface guiding a user through the generation of each FOS, including creation of linked FOSs including one or more of a safety FOS and a general guidelines FOS.

12. The method of claim 1, further comprising presenting an interactive interface guiding a user through the generation of each FOS, including validation conditions that include validation by one or more of machine vision, similarity algorithms, rule based, semantic context based generation, machine, deep learning, data augmentation and simulation, and algorithms.

13. The method of claim 1, further comprising presenting an interactive interface guiding a user through the generation of each FOS, including a top-down, bottom-up, or singleton creation process.

14. The method of claim 1, further comprising presenting an interactive interface prompting a user to choose an engineering procedure type for a FOS to be generated and guiding the user through a FOS creation process according to the procedure type.

15. The method of claim 1, further comprising presenting an interactive interface prompting a user to choose conditions for step validation with input having a form of one or more of: a “check” type to select a yes/no response, a color response, a numeric or alphanumeric response, a range of values that can be integer or floating point numbers, a picture to be matched, and a video to be analyzed.

16. The method of claim 1, further comprising presenting an interactive interface prompting a user to make graphical additions to the BOM image including one or more of additional text, images, models, balloons, and graphical signs.

17. The method of claim 1, wherein each part and each FOS is definable as a global component or a local component, wherein components are accessible by one or more additional interactive production portfolios.

18. The method of claim 1, further comprising checking completeness of all FOSs by confirming that all parts having sub-parts have one or more associated FOSs and that all

FOSs include only parts in the hierarchical list.

19. The method of claim 1, further comprising presenting an interactive interface prompting a user to define triggering events for FOS execution including a time event and/or a fault event.

20. The method of claim 1, wherein a type of FOS is generated as a training procedure to present steps to be learned by a user, that, when presented, provide certification for a specific task or real-time assistance to a user.

21. The method of claim 1 further comprising presenting an interactive interface providing a graphical measure of a FOS status including at least a status from the following status list: 1) procedure lacking content; 2) detected errors in procedure; 3) validated procedure; 4) approved procedure.

22. The method of claim 1, further comprising logging, to a log file, parameters of user interaction with an interactive interface for generating the interactive production portfolio, including recording of user actions, system actions, time stamps, tool usage, jigs, materials, specific combinations of procedures, FOSs and steps, to standardize and improve an engineering knowledge base, and wherein the log file identifies steps and procedures that are incomplete, unvalidated, unapproved, erroneous, and/or conflicting.

23. The method of claim 1, wherein the interactive production portfolio provides content reusability of projects, parts, procedures, and steps, updating of content, managing changes, comparing project versions, and propagating changes while requesting pre-defined, necessary user approval for creation of a new project version.

24. A system for generating an interactive production portfolio for manufacturing and/or maintaining a product comprising a computer processor having associated non-transient memory with instructions that when executed implement steps including: (i) receiving a plurality of engineering documents and computer-aided design (CAD) models, including a hierarchical list of parts and their sub-parts, and hierarchical images of the parts and their sub-parts;

(ii) for each part with sub-parts, storing in the production portfolio a part record linked to respective sub-part instance records, wherein each sub-part instance record in turn links to a part record;

(iii) linking to each part record in the production portfolio a link to one or more flows of steps (FOSs), wherein a step, for at least one of assembling or disassembling the part from its sub-parts, indicates: 1) one or more sub-parts involved in the given step, indicated by a bill of materials (BOM) image and/or by text, 2) a validation condition for determining a valid or invalid result of the step, 3) a next step to perform for the valid result and an alternative step to perform for the invalid result;

(iv) for each part record, verifying that each linked sub-part instance record is also indicated as a sub-part in a step of the linked FOS, and vice-versa, that each sub-part linked to the steps of the linked FOS is also linked as a sub-part instance record to the part record;

(v) presenting steps for manufacturing or maintenance of a part of the portfolio in a graphical, step-by-step manner by accessing the linked FOS of the part, including the validation condition for each step result of each step, receiving a result, determining conformance with the step’s condition, and presenting the step’s next step when the result is valid and the step’s alternative step when the result is invalid.