HK1254084B - System and methods for auditing a virtual machine - Google Patents

Description

System and method for auditing virtual machines

Related applications

This application claims the benefit of the filing date of U.S. Provisional Patent Application No. 62/274,902, filed on January 5, 2016, entitled “Systems and Methods for Auditing a Virtual Machine,” which is incorporated herein by reference in its entirety.

Background Art

The present invention relates to systems and methods for performing software auditing, and more particularly, to software auditing performed in hardware virtualization configurations.

In the modern software-driven economy, the volume and complexity of hardware and software assets held by companies continues to increase. As employees transition from using desktop computers to using mobile devices (such as laptops, tablets, and mobile phones), and as work becomes more location-independent, keeping track of a company's hardware and software assets becomes a significant problem. For example, instead of using a single version of a software application across all company computers, companies now typically use multiple versions of each application to account for the various hardware platforms and operating systems used by employees in the modern mobile office. This situation is further complicated when software updates are applied, as such updates may not be delivered consistently to all computers and mobile devices.

This increased heterogeneity can complicate software licensing and service-level agreements. Furthermore, the recent proliferation of application software for mobile devices creates a growing computer security risk. Malware and spyware can easily disguise themselves among the plethora of legitimate applications and versions, exposing employees and companies to the risk of unauthorized access to proprietary data and lost productivity.

Another recent development that has altered the classic computing model is hardware virtualization. In applications such as web server farms and virtual desktop infrastructure (VDI), hundreds of virtual machines can run simultaneously on a single physical platform. These virtual machines can be dynamically instantiated and/or removed, further increasing the heterogeneity of the software running on a given physical platform at any given time.

Audit software can be used to record the software installed and/or currently executing on a computer system for purposes such as licensing, digital rights management, application control, and computer security.There is growing interest in developing audit systems and methods that are particularly well-suited for modern virtualized environments.

Summary of the Invention

According to one aspect, a computer system includes at least one hardware processor configured to execute a set of client virtual machines (VMs) and further execute a VM audit engine external to the set of client VMs. The VM audit engine is configured, in response to receiving an audit request from a remote audit server, to insert an audit agent into a target VM of the set of client VMs, the audit agent being configured to perform an audit of the target VM, the audit including items selected from the group consisting of: generating a list of legal computer programs installed for execution on the target VM; and determining the amount of hardware resources currently used by the target VM. The VM audit engine is further configured, in response to inserting the audit agent, to cause the target VM to execute the audit agent; and in response to the target VM executing the audit agent, to remove the audit agent from the target VM.

According to another aspect, a method includes executing a virtual machine audit engine externally to a group of client VMs executing on the computer system using at least one hardware processor of a computer system. Executing the VM audit engine includes: in response to receiving an audit request from a remote server, inserting an audit agent into a target VM of the group of client VMs, the audit agent being configured to perform an audit of the target VM, the audit including items selected from the group consisting of: generating a list of legal computer programs installed for execution on the target VM; and determining an amount of hardware resources currently used by the target VM. Executing the VM audit engine further includes: in response to inserting the audit agent, causing the target VM to execute the audit agent; and in response to the target VM executing the audit agent, removing the audit agent from the target VM.

According to another aspect, a server computer system includes at least one hardware processor configured to perform an audit transaction using multiple client systems. The audit transaction includes: sending an audit request to a client system of the multiple client systems; and in response, receiving an audit report from the client system, the audit report being determined by a virtual machine VM audit engine executing on the client system outside a group of client VMs executing on the client system. Determining the audit report includes: in response to receiving the audit request from the server computer system, inserting an audit agent into a target VM of the group of client VMs, the audit agent being configured to perform an audit of the target VM, the audit including items selected from the group consisting of: generating a list of legal computer programs installed for execution on the target VM; and determining the amount of hardware resources currently used by the target VM. Determining the audit report further includes: in response to inserting the audit agent, causing the target VM to execute the audit agent; and in response to the target VM executing the audit agent, removing the audit agent from the target VM.

According to another aspect, a non-transitory computer-readable medium storing instructions that, when executed by at least one hardware processor of a computer system, cause the computer system to form a virtual machine audit engine, the VM audit engine executing externally to a set of client VMs disclosed on the computer system, wherein the VM audit engine is configured, in response to receiving an audit request from a remote audit server, to: insert an audit agent into a target VM in the set of client VMs, the audit agent configured to perform an audit of the target VM, the audit including items selected from the group consisting of: generating a list of legal computer programs installed for execution on the target VM; and determining an amount of hardware resources currently used by the target VM. The VM audit engine is further configured, in response to inserting the audit agent, to cause the target VM to execute the audit agent; and in response to the target VM executing the audit agent, to remove the audit agent from the target VM.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing aspects and advantages of the present invention will be better understood after reading the following detailed description and referring to the accompanying drawings, in which:

1 illustrates an exemplary configuration in which multiple client systems are audited at the request of an audit server according to some embodiments of the present invention.

FIG2-A illustrates an exemplary hardware configuration of an audited client system according to some embodiments of the present invention.

FIG2-B illustrates an exemplary hardware configuration of an audit server according to some embodiments of the present invention.

3-A shows an exemplary virtual machine exposed by a hypervisor executing on an audited client system and a VM audit engine executing external to the audited VM, according to some embodiments of the invention.

3-B illustrates an alternative configuration in which guest virtual machines are audited by a separate audit VM executing on the client system, according to some embodiments of the invention.

4 shows an exemplary sequence of steps performed by an audit installation application to set up auditing of a client system, according to some embodiments of the invention.

FIG5 shows an exemplary data exchange between an audited client system and an audit server according to some embodiments of the present invention.

FIG6 illustrates an exemplary sequence of steps performed by a VM audit engine according to some embodiments of the invention.

7 shows exemplary components of an audit agent executing at various processor privilege levels according to some embodiments of the invention.

8 shows an exemplary sequence of steps performed by a VM audit engine to place an audit agent into an audited VM, according to some embodiments of the invention.

9 illustrates an exemplary sequence of steps performed by a driver loader of an audit agent according to some embodiments of the present invention.

FIG. 10 shows an exemplary sequence of steps performed by an audit driver according to some embodiments of the present invention.

DETAILED DESCRIPTION

It should be understood that in the following description, all connections between structures may be direct operational connections or indirect operational connections through intermediate structures. A group of elements includes one or more elements. It should be understood that any statement of an element refers to at least one element. A plurality of elements includes at least two elements. Unless otherwise required, any method steps described do not necessarily need to be performed in the specific order described. A first element (e.g., data) derived from a second element encompasses a first element that is equal to the second element, and also encompasses a first element generated by processing the second element and optionally other data. Making a determination or decision based on a parameter encompasses making a determination or decision based on the parameter and optionally other data. Unless otherwise specified, an indicator of some quantity/data may be the quantity/data itself or an indicator different from the quantity/data itself. A computer program is a sequence of processor instructions that implements a task. The computer program described in some embodiments of the present invention may be an independent software entity or sub-entity (e.g., a subroutine, a library) of another computer program. Unless otherwise specified, a legitimate computer program is a computer program that is installed or otherwise configured to be executed by a legitimate user of the corresponding computer system. In contrast, malicious software (malware) and computer programs installed by unauthorized intruders on a corresponding computer system are considered illegitimate computer programs herein. A computer program is said to execute within or inside a virtual machine (or the corresponding virtual machine executes the corresponding computer program) when the corresponding computer program is executed on at least one virtual processor of the corresponding virtual machine. A process is an instance of a computer program (e.g., an application) or a portion of an operating system, characterized by having at least one thread of execution and a virtual memory space assigned to it, wherein the contents of the corresponding virtual memory space contain executable code. Unless otherwise specified, a page represents the smallest unit of virtual memory that can be individually mapped to the physical memory of a host system. Unless otherwise specified, a register represents a storage component that is integrated with or forms part of a processor and is distinct from random access memory (RAM). A tunnel is a virtual point-to-point connection between two entities connected to a communication network. Computer-readable media encompasses non-transitory media such as magnetic, optical, and semiconductor storage media (e.g., hard drives, optical disks, flash memory, DRAM), as well as communication links such as conductive cables and fiber optic links. According to some embodiments, the present invention provides, among other things, a computer system comprising hardware (eg, one or more microprocessors) programmed to perform the methods described herein, and a computer-readable medium encoding instructions to perform the methods described herein.

The following description illustrates embodiments of the invention by way of example and not necessarily by way of limitation.

FIG1 shows an exemplary configuration according to some embodiments of the present invention, in which a plurality of client systems 12a-d are audited remotely at the request of an audit server 14. Exemplary client systems 12a-d include personal computer systems, mobile computing platforms (laptops, tablets, mobile phones), entertainment devices (TVs, game consoles), wearable devices (smart watches, fitness bands), home appliances, and any other electronic device that includes a processor and memory and is capable of operating a hardware virtualization platform. Client systems 12a-d are interconnected via a communication network 10, such as an enterprise network, the Internet, etc. Components of network 10 may include a local area network (LAN).

An audit server 14 is communicatively coupled to the client systems 12a-d and cooperates with each client system 12a-d to perform an audit of the respective client system. Exemplary audit activities include, among other things, identification of software assets and verification of the respective software assets with respect to licensing, usage, and rights. The granularity of such audits can vary from relatively high-level (e.g., determining which applications/versions are installed on the client system) to in-depth audits (including, for example, determining whether the software executing on the client system uses specific versions of key components (e.g., drivers, patches, anti-malware modules), when each component was installed or accessed, how many hardware resources a single application uses, etc.). Other exemplary audit activities include, among other things, determining which software components (applications, processes, drivers, etc.) are currently executing on the client system at the time of the audit, determining the current state of the respective client system's processor, determining current resource usage levels (e.g., CPU, memory, disk, network, etc.), and determining a set of configuration parameter values/settings for the respective client system's OS, various applications, user settings, etc.

The server 14 generally represents a group of interconnected computing systems that may or may not be in physical proximity to one another. In some embodiments, the audit server 14 is configured to access a client database 15. In the exemplary client database 15, each entry is associated with an audited client system 12a to d and may include the contents of a set of audit reports (see below) received from the respective client system. Each entry may further be time-stamped with an indicator of the time each audit report received from the respective client system was received. Each entry in the database 15 may include audit data determined for the respective client system, including, for example, an indicator of the hardware configuration of the respective client system, an indicator of the type and version of the operating system (OS), an indicator of a set of legally installed applications or other software components (e.g., drivers, patches), an indicator of a set of software components (applications, processes, drivers, etc.) currently loaded and/or executing at the time of the audit, a set of parameters indicating various OS, application, and/or user settings of the respective client system, and a set of indicators of resource usage (e.g., CPU, memory, disk, etc.) at the time of the audit. The audit data may further include a time indicator indicating when each listed software component was installed and/or when each listed component was last accessed or modified.

FIG2-A shows an exemplary hardware configuration of a client system 12 (e.g., systems 12a to d in FIG1 ). For simplicity, the client system illustrated is a computer system; the hardware configuration of other client systems (e.g., mobile phones, watches, etc.) may differ slightly from the illustrated configuration. Client system 12 includes a set of physical devices, including a hardware processor 16 and a memory unit 18. Processor 16 includes a physical device (e.g., a microprocessor, a multi-core integrated circuit formed on a semiconductor substrate, etc.) configured to perform computing operations and/or logical operations using a set of signals and/or data. In some embodiments, such operations are delivered to processor 16 in the form of a sequence of processor instructions (e.g., machine code or other type of encoding). Memory unit 18 may include volatile computer-readable media (e.g., DRAM, SRAM) that stores instructions and/or data accessed or generated by processor 16.

Input devices 20 may include a computer keyboard, mouse, and microphone, among others, including corresponding hardware interfaces and/or adapters that allow a user to enter data and/or instructions into client system 12. Output devices 22 may include display devices such as a monitor and speakers, and may also include hardware interfaces/adapters that allow client system 12 to communicate data to the user, such as a graphics card. In some embodiments, input devices 20 and output devices 22 may share a common piece of hardware, such as in the case of a touchscreen device. Storage devices 24 include computer-readable media that enables non-volatile storage, reading, and writing of software instructions and/or data. Exemplary storage devices 24 include magnetic and optical disks and flash memory devices, as well as removable media such as CD and/or DVD disks and drives. A set of network adapters 26 enables client system 12 to connect to a computer network and/or to other devices/computer systems. Controller hub 28 represents multiple system, peripheral, and/or chipset buses and/or all other circuitry that enables communication between processor 16 and devices 18, 20, 22, 24, and 26. For example, controller hub 28 may include a memory controller, an input/output (I/O) controller, an interrupt controller, etc. In another example, controller hub 28 may include a north bridge that connects processor 16 to memory 18 and/or a south bridge that connects processor 16 to devices 20, 22, 24, and 26.

FIG2-B shows an exemplary hardware configuration of an audit server 14 according to some embodiments of the present invention. Server 14 includes at least one hardware processor 116 (e.g., a microprocessor, a multi-core integrated circuit), physical memory 118, and a set of server network adapters 126. Adapters 126 may include network cards and other communication interfaces that enable audit server 14 to connect to communication network 10. In some embodiments, server 14 further includes input, output, and storage devices, which may be functionally similar to input 20, output 22, and storage device 24, respectively.

In some embodiments, client system 12 is configured to expose a set of virtual machines, such as those illustrated in Figures 3-A to 3-B. Virtual machines (VMs) emulate actual physical machines/computer systems using any of a variety of techniques known in the art of hardware virtualization. In some embodiments, a hypervisor 30 (also known in the art as a virtual machine monitor (VMM)) executes on client system 12 and is configured to create or enable multiple virtualized devices, such as virtual processors and virtual memory management units, and expose such virtualized devices to software instead of the actual physical devices of client system 12. Such operations are generally known in the art as exposing virtual machines. Hypervisor 30 can further enable multiple virtual machines to share the hardware resources of host system 12, such that each VM operates independently and is unaware of other VMs currently executing on client system 12. Examples of popular hypervisors include VMware vSphere ^™ from VMware Inc. and the open source Xen hypervisor, among others.

In the exemplary configuration illustrated in Figures 3-A to 3-B, a guest VM 32 executes a guest operating system (OS) 34 and a set of applications 36a to b. Guest OS 34 may include any widely available operating system, such as Microsoft Windows or the like, which provides an interface between applications executing within VM 32 and the virtualized hardware devices of VM 32. Applications 36a to b generally represent any user application, such as a word processor, spreadsheet application, graphics application, browser, social media and electronic communication application, etc.

3-A , VM audit engine 40 a executes outside guest VM 32 at the processor privilege level (e.g., root level or ring-1) of hypervisor 30. VM audit engine 40 a is configured to perform an audit of guest VM 32. In some embodiments, this audit includes placing an audit agent 42 in the audited VM, as described in detail below, and removing the agent 42 from the audited VM once the audit is complete.

In an alternative embodiment illustrated in FIG3-B , hypervisor 30 is configured to expose an audit VM 33, separate from all audited guest VMs, where audit VM 33 can execute concurrently with guest VM 32. Some embodiments of audit VM 33 include a lightweight minimal operating system (e.g., a customized version of an OS) and are configured to execute a VM audit engine 40b, which is configured to perform an audit of guest VM 32. The virtual environments of guest VM 32 and audit VM 33 can be isolated from each other using any method known in the virtualization art to ensure that malware executing within guest VM 32 cannot infect or otherwise interfere with software executing within audit VM 33. In the embodiment of FIG3-B , introspection engine 44 executes externally to guest VM 32 and audit VM 33 at the processor privilege level of hypervisor 30. In the exemplary embodiment of FIG3-A , some of the activities of introspection engine 44 can be performed, at least in part, by VM audit engine 40a.

Although Figures 3-A to B are drafted to show only one client VM 32 executing on a client system 12, some embodiments may be configured to perform audits on client systems running multiple VMs. A typical application for such a virtualization platform is a server farm, where a single computer system may manage hundreds of different web server VMs operating simultaneously. Another typical application falls into the category of virtual desktop infrastructure (VDI), where a user's application executes on a first computer system while the user interacts with the corresponding application via a second computer system (terminal). In a typical VDI configuration, virtual machines running requested applications are instantiated on demand on the first computer system, which can result in hundreds of such VMs being executed for multiple remote users. In such embodiments, a single VM audit engine 40a to b may perform audits of multiple VMs or all VMs executing on the corresponding client system. For example, the audit engine 40a to b may place an audit agent inside each of the corresponding client VMs. In one such example, VM audit engines 40a-b may select a type of audit agent and adjust the agent delivery procedures based on the hardware and/or software specifications of each VM (e.g., based on the type of OS executing within the respective VM). In some embodiments, VM audit engines 40a-b may be directed by server 14 to selectively perform audits of specific target VMs or a selected subset of VMs executing on the respective client systems.

In some embodiments, introspection engine 44 ( FIG. 3-B ) is configured to detect various events that occur during execution of software within an audited guest VM and to exchange this information with other auditing software (e.g., agent 42 and/or engines 40 a-b). Exemplary events detected by introspection engine 44 include, for example, processor exceptions and/or interrupts, attempts to execute specific functions of guest OS 34, changes in processor privileges (e.g., system calls), attempts to access (read from, write to, and/or execute from) specific memory locations, etc. Introspection engine 44 may further be configured to determine the memory addresses of various software components executing within guest VM 32, as further described below.

Several methods for detecting such events are known in the art. In one such example, the introspection engine 44 may cooperate with the hypervisor 30 to set memory access permissions using a second-level address translation mechanism (e.g., Extended Page Tables (EPT) on the platform, Rapid Virtualization Indexes (RVI) on the platform). When an attempt to access a particular memory page violates the access permissions set for the corresponding memory page, the corresponding attempt may trigger a processor event, such as an exception or a virtual machine exit event (VMExit on the platform). As a result of the processor event, the processor 16 may switch to executing an event handler routine within the introspection engine 44, which allows the engine 44 to detect the occurrence of the corresponding event. Such a mechanism can detect, for example, a call to a specific function by marking the memory page containing a portion of the corresponding function as non-executable.

For the sake of simplicity and without loss of generality, the following description will focus on an exemplary embodiment configured as illustrated in FIG3-A . Those skilled in the art will appreciate that the described systems and methods can be adapted to other configurations, and in particular, to the exemplary configuration of FIG3-B . In the following description, the VM audit engine will be generally designated 40 , which may represent either audit engine 40 a or audit engine 40 b , depending on the configuration chosen.

FIG4 illustrates an exemplary sequence of steps performed to configure an audit of a client system, according to some embodiments of the present invention. In a typical scenario for auditing an enterprise network, a network administrator may install an audit application on each client system 12a-d. The audit application may include various components, such as a VM audit engine 40, an introspection engine 44, and the like. The illustrated sequence of steps may be implemented, for example, by an installer utility for the respective audit application. When installed on a client system not currently operating in a hardware virtualization configuration, the audit software may first take over the processor 16 at the highest privilege level (e.g., VMXRoot on a platform that supports virtualization, also commonly referred to as root mode or ring-1) and install the hypervisor 30. The hypervisor 30 may then expose the guest VM 32 and move all software previously executed on the respective client system to execute within the guest VM 32. The hypervisor 30 may further configure the VM audit engine 40 and/or the introspection engine 44. When installed on a hardware virtualization platform that is already running multiple virtual machines, steps 200-202 may be omitted.

In some embodiments, step 206 may set up remote access from audit server 14 to VM audit engine 40. This access may enable the audit server to send instructions directly to audited client system 12, either automatically or with the assistance of a human operator. Such instructions may, for example, direct VM audit engine 40 to perform a particular type of audit, verify the presence of a guest VM and/or the characteristics of a particular software component, determine the current state of processor 16 (e.g., the contents of a set of processor registers), read a set of parameter settings for guest OS 34 or other software, etc. In one exemplary embodiment, step 206 may configure a tunnel (i.e., a point-to-point communication link) between server 14 and hypervisor 30 and/or between server 14 and audit VM 33. Exemplary tunnels may be set up according to virtual private network (VPN) and/or secure shell (SSH) protocols and methods. Once this tunnel is set up, it may be used by server 14 to send audit requests and/or other instructions to VM audit engine 40.

In another step 208, the audit installer may configure introspection and/or event interception from the hypervisor 30 level. Step 208 may include, for example, setting access permissions for certain memory pages used by the audited guest VM so that an attempt to access the corresponding page will trigger a processor event (e.g., a VM exit event), thereby enabling the VM audit engine 40 or introspection engine 44 to detect the occurrence of an event (e.g., a system call) within the audited guest VM 32. In some embodiments, step 208 may further set up a signaling and/or communication mechanism between components executing within the audited VM (e.g., audit agent 42) and components executing outside the corresponding VM (e.g., audit engine 40, introspection engine 44). This communication or signaling may use any method known in the art of hardware virtualization. In one example, two software components may transfer data between each other via a physical memory segment shared by the two software components. To send data from a first component executing within the VM to a second component executing outside the corresponding VM, the first software component may write the data to the shared memory segment and then issue a privileged processor instruction (e.g., VMCALL on the platform), thereby causing a VM exit event. A VM exit event (e.g., VMExit on the platform) switches the processor 16 from executing the first software component to executing a handler routine (which may be part of a second software component). Thus, a second component executing external to the respective VM can be notified that the first component is attempting to transfer data. Conversely, to send data from the second component to the first component, the second component can write the data to a shared memory segment and then inject an interrupt into the respective VM. The interrupt will be handled by an interrupt handler executing within the respective VM (e.g., handled by the first software component). Thus, the first component can detect that the second component is attempting to transfer data. In some embodiments, step 208 includes registering the appropriate software component as a handler for various processor events to enable the communication mechanism described above.

FIG5 illustrates an exemplary data exchange between audit server 14 and an audited client system according to some embodiments of the present invention. Server 14 may send an audit request 50 to client system 12, directing VM audit engine 40 to perform an audit of client system 12. In some embodiments, audit request 50 includes an indicator of the audit type (e.g., an indicator of the type of audit operation to be performed on client system 12), which is used to list all legitimate applications, determine current OS settings, and the like. Another exemplary audit request 50 includes an indicator of the target virtual machine currently executing on client system 12; for example, the audit request may direct audit engine 40 to selectively perform an audit of the target VM. Yet another exemplary audit request 50 may include an indicator of a specific software component (e.g., a Microsoft-specific driver, etc.). This request may direct engine 40 to determine, among other things, whether the corresponding software component is legitimately installed on the corresponding client system, which version of the corresponding software component is installed, whether the corresponding software component is currently loaded in memory/currently executing, when the corresponding software component was most recently used, and the like. Another exemplary audit request 50 may direct the engine 40 to determine the current level/amount/percentage of hardware resources (eg, processor, storage, memory) used by the target VM or a particular software component executing within the target VM.

Audit request 50 may include an identifier of a target customer VM executing on the audited client system. This request may direct audit engine 40 to perform a selective audit of the corresponding customer VM. Other requests 50 may direct audit engine 40 to perform an audit of all customer VMs executing on the audited client system.

Audit requests 50 may be sent according to a schedule or in response to an action by a network administrator. Audit requests 50 may be sent uniformly to all client systems 12a-d or may be tailored specifically to each audited client system. In some embodiments, audit requests may be triggered by an event occurring on the corresponding client system or in response to an event occurring anywhere on the communication network 10. In response to performing the requested audit, VM audit engines 40a-b may send an audit report 52 to server 14. Report 52 includes the results of the corresponding audit activity, such as a list of installed software components, a set of current configuration settings for the corresponding client system, and the like.

6 shows an exemplary sequence of steps performed by VM audit engine 40 according to some embodiments of the present invention. Engine 40 may listen for audit requests 50 from server 14. Upon receiving request 50, engine 40 may select a target VM for auditing based on audit request 50 and place an audit agent in the target client VM (e.g., client VM 32 in FIG. 3-A-B ).

An agent may be drafted based on the type of the audited guest VM, for example, based on the type of guest OS 34 currently executing within the audited VM. The OS type may include a name indicator (e.g., version 7, Home Edition, or Enterprise Edition), among others. In some embodiments, the audit engine 40 identifies the OS type based on the contents of the corresponding guest VM's model-specific registers (MSRs) or the contents of the memory segments pointed to by the corresponding MSRs. In some embodiments, the engine 40 may determine the OS name based on data written to such MSRs by software executing within the audited VM. For example, the engine 40 may intercept instructions written to the SYSENTER or SYSCALL MSRs and determine the type of the currently executing or currently instantiated OS based on the parameters of the written instructions. Other exemplary registers that may provide information about the OS name include control registers, the Interrupt Descriptor Table (IDT), and the Global Descriptor Table (GDT). To identify the OS type based on MSR writes, the introspection engine 40 may further utilize pattern matching with a library of predefined fast system call handlers specific to each OS (e.g., system calls handled based on the contents of the SYSCALL or SYSENTER MSRs). Such a fast system call library may be provided with an audit engine 40 and/or an introspection engine 44 and may be kept up to date via periodic or on-demand software updates.

In some embodiments, the version indicator (e.g., release name, build number, etc.) can be obtained by parsing certain kernel data structures of the corresponding type specific to the OS. An exemplary data structure that allows identification of the OS version is certain exported symbols of the kernel or certain exported symbols of the kernel, such as NtBuildNumber, etc.

The audit agent 42 is placed in the target client VM, and the sequence of steps 228 to 230 to 232 can be repeated in a loop until the audit is complete. While the audit is in progress, the VM audit engine 40 can exchange data with the audit agent 42 (e.g., memory addresses of various software components, various results of the audit). This messaging between components executing within the virtual machine and components executing outside the virtual machine can be implemented using any method known in the art of virtualization (e.g., via a memory segment shared between the audit engine 40 and the audit agent 42, as described above). When the audit is complete, step 234 erases the audit agent from the audited VM, for example, by erasing the memory segment containing the driver loader (described in more detail below). In another step 236, the audit engine 40 formulates an audit report 52 and sends the audit report 52 to the audit server 14.

FIG7 illustrates exemplary components of audit agent 42 from the perspective of a processor privilege level (e.g., a privileged ring). In some embodiments, audit agent 42 includes a driver loader 46 and an audit driver 48. Driver loader 46 can be injected by VM audit engine 40 into a running application 36, such as applications 36a-b in FIG3-A-B. Driver loader 46 can therefore execute at the privilege level of application 36, typically in ring 3 (user mode or user processor privilege level). In some embodiments, driver loader 46 loads and starts audit driver 48 on guest VM 32, typically executing at the processor privilege level of guest OS 34 (e.g., ring 0, kernel mode, or kernel level). Audit driver 48 can then perform audit operations within guest VM 32. When the audit is complete, loader 46 or VM audit engine 40 can remove driver 48 from guest VM 32, for example, by erasing the memory segment containing driver 48. Alternative embodiments do not use driver loader 46, but instead inject code (e.g., audit agent 42) directly into the kernel of guest OS 34. In such embodiments, agent 42 may execute entirely in ring 0 (kernel mode).

Placing software components into a virtual machine can be accomplished using various methods known in the art. FIG8 shows an exemplary sequence of steps performed by VM audit engine 40 to place agent 42 inside guest VM 32 (step 226 in FIG6 ), according to a preferred embodiment of the present invention. In step 250, VM audit engine 40 identifies a memory segment suitable for injecting driver loader 46. In some embodiments, driver loader 46 is drafted to be as small as possible, e.g., much smaller than a memory page, so that it can be safely disguised within existing data structures without requiring the use of the OS's memory allocation facilities (use of which makes loader 46 and/or driver 48 discoverable by other software, including malware, executing within guest VM 32).

In one example, VM audit engine 40 may search for a memory segment suitable for receiving driver loader 46 within the padding space typically reserved by guest OS 34 between driver segments. This padding space exists because some operating systems, for example, allocate memory so that each different segment of an object is aligned with a page boundary. When a segment does not occupy an entire memory page, the remaining memory space within the corresponding page is filled with dummy data (e.g., zero values). In such embodiments, searching for suitable memory space to accommodate driver loader 46 may include scanning memory pages allocated to the driver of guest OS 34 for this dummy data.

In another example, the engine 40 may intercept an attempt by the OS to allocate memory for a small object (e.g., a driver), where the corresponding object is smaller than a memory page. Instead of allocating the required amount of memory, the engine 40 may force the guest OS 34 to allocate a full memory page (e.g., 4kB) to the corresponding object and use the remaining space within the corresponding memory page to control the driver loader 46. To intercept such memory allocation attempts, some embodiments of the audit engine 40 and/or introspection engine 44 may detect attempts to execute native OS memory management functions, such as KeAllocatePoolWithTag in . To determine the memory address where such functions reside in the memory of the corresponding guest virtual machine 32, the audit engine 40 may access certain data structures, such as the exported function table of the kernel binary image (e.g., Portable Executable Format in , Executable and Linkable Format in ). The type of object currently being allocated can be determined based on the allocation tag of the intercepted call. For example, in , the 'Driv' tag indicates a driver object.

In another example, step 250 includes mapping unused memory pages into the memory space of the executing process (e.g., application 36 in FIG7 ). Having identified the appropriate memory segment for host driver loader 46, in step 252, VM audit engine 40 may inject loader 46 into the corresponding memory segment.

Next, some embodiments of the audit engine 40 may wait for the currently executing process (e.g., application 36) to attempt a processor privilege change, for example, by issuing a system call (e.g., issuing SYSCALL on the platform, or issuing SYSENTER on the platform). Intercepting system calls from outside the corresponding VM can be accomplished in several ways. In one example, the engine 40 may reset the values of processor registers of the virtual processor of the currently executing thread attempting the system call. Examples of such processor registers include the IA32_LSTAR and IA32_SYSENTER_EIP registers on the platform. This register manipulation will subsequently result in a fault when the system call is attempted; the corresponding fault can be detected by the VM audit engine 40 or the introspection engine 44. In another example, the engine 40 may place a redirection connection on the code page containing the system call handler of the guest OS 34.

In some embodiments, when system call is intercepted, step 256 implements one group of qualified inspection, to determine whether the context of corresponding system call allows the safe injection of driver loader 46.In an example in which the interception of system call is realized via the connection of system call handler, qualified inspection can include determining whether current stack is user stack. When stack is user stack, attempting to inject code may endanger system, because anything resident in user mode is all that kernel is untrusted. In addition, other threads can peek at the data from user stack, and can even be able to even attack driver loader 46. In some embodiments, determining whether stack is user stack comprises determining the privilege level of stack fragment descriptor. Another qualified inspection can determine whether interrupt is currently enabled. Driver loader 46 may have to perform some processor instructions that need to enable interrupt. Another qualified inspection can determine current interrupt request level (IRQL). Some embodiments may need (for example) current IRQL equal to 0 to be able to implement the operation of driver loader 46.

When the system call meets the eligibility requirements, in step 260, audit engine 40 may suspend execution of the thread that issued the corresponding system call and switch processor 16 to execute driver loader 46. When loader 46 has completed execution, audit engine 40 may switch processor 16 back to executing the original thread (e.g., application 36).

Figure 9 shows an exemplary sequence of steps performed by the driver loader 46 according to some embodiments of the present invention. In order to allocate memory for the audit driver 48 and/or to create a thread for the driver 48, some embodiments may call specialized functions of the guest OS 34, such as ExAllocatePoolWithTag and PsCreateSystemThread in (similar functions exist in other OSes (e.g., )). In embodiments where it is desired that the driver loader 46 have a small memory footprint, the addresses of such OS functions may be provided to the loader 46 by the VM audit engine 40, for example, via the messaging mechanism described above. In such embodiments, the driver loader 46 may not actually write the driver 48 to the allocated memory. Instead, the loader 46 may communicate the address of the allocated memory segment to the VM audit engine 40, with the actual write being performed by the audit engine 40.

Once audit driver 48 is started (step 274), loader 46 may exit. In some embodiments, when audit driver 48 completes execution, e.g., when the current audit operation is complete, driver loader 46 may remove audit driver 48 from guest VM 32. In alternative embodiments, VM audit engine 40 may perform cleanup (e.g., remove driver 48 from memory).

FIG10 illustrates exemplary steps performed by audit driver 48 according to some embodiments of the present invention. Because driver 48 executes within guest VM 32, which has its own memory space and execution thread, driver 48 can use all resources available to guest OS 34 to perform audits of guest VM 32, such as enumerating a list of installed applications, determining the values of various OS parameters, determining time stamps associated with various software components, determining resource usage, and so on. Such tasks can be implemented using various methods known in the art. For example, audit driver 48 can enumerate certain registry keys of guest OS 34, which reveal the identities of installed applications/software packages, etc. Another auditing method calls an application programming interface (API) exposed by guest OS 34 (e.g., Windows Management Instrumentation (WMI) in Windows) to obtain various data about installed programs. In yet another example, driver 48 can search specific files and folders (e.g., program files in Windows) to obtain a list of installed programs and updates. These exemplary methods can be used individually or together. Data collected during the audit can be transmitted to VM audit engine 40, for example, by writing this data to a predetermined memory segment shared between driver 48 and VM audit engine 40.

In some embodiments, in response to performing an audit and before exiting, the audit driver 48 may load and/or start a security module, such as an anti-malware driver. The security module may determine, for example, whether the audited guest VM includes malware and/or may delete existing malware or otherwise disable existing malware. Due to the stealthy and dynamic deployment of this security module, its presence and activity may be relatively difficult to detect and may be interfered with by malware executing within the corresponding guest VM.

The exemplary systems and methods described herein allow for remotely performing software audits on a relatively large number of client systems (e.g., an enterprise network). Software auditing, as used herein, refers to evaluating legitimate software, i.e., evaluating software that has been installed and/or configured for execution on the respective client systems by legitimate users (e.g., system administrators) of the respective client systems. The exemplary audit can determine, for example, which software applications are currently installed on the company's computers. This information can be used, for example, to facilitate software monitoring, deploy upgrades, prevent employees from installing or using certain software (e.g., chat applications, games) at work, determine that employees are using a particular brand of software (e.g., ), and the like.

Audit results can be conveniently centralized at a server computer communicatively coupled to the audited client systems. Audits can be automatically performed at the request of a human administrator, according to a schedule, and/or on demand. In some embodiments of the present invention, audits can be further coupled with computer security activities, such as targeted malware scanning/disinfection of selected client systems.

Some embodiments use hardware virtualization technology to improve the security and reliability of software audits. To avoid exposing the audit software to malicious human intervention and/or malware infecting the audited client, some embodiments move the client software, including the operating system, into a virtual machine (VM) and execute the audit portion from outside the respected VM. Portions of the audit software can thus execute below the hypervisor level or within a separate, dedicated audit VM executing on the respective client.

By taking full advantage of virtualization, some embodiments of the present invention are also well suited for auditing client systems that execute multiple VMs in parallel, such as server farms and virtual desktop infrastructure (VDI) systems. A single audit engine configured in accordance with some embodiments can audit multiple VMs or all VMs executing on a corresponding client. Some conventional systems configured to implement audit operations in a virtualized environment perform audits by analyzing memory snapshots of the audited VM. Such conventional systems may require that the audited VM be stopped in order to take the corresponding memory snapshot. In contrast, some embodiments of the present invention do not require the audited VM to be stopped because the audit agent can be placed into the currently executing VM. By not being limited to the information that can be extracted from the memory snapshot, some embodiments may be able to access substantially richer audit data from within the executing VM.

Permanently installing audit software within an audit client may expose the software to malware, which may stop the software or otherwise prevent it from functioning properly. Compared to such conventional solutions, some embodiments of the present invention dynamically place an audit agent within the audited machine for the duration of the audit, thereby minimizing computer security risks. Another advantage of not having a permanent audit agent within the audited VM is that such a configuration may significantly facilitate the management, deployment, and upgrade of both the virtual machine and the audit software. When the audit solution is standalone and not permanently dependent on the audited virtual machine, the audit software can be updated independently of the VM software. For example, an update to the VM audit engine 40 can be installed without affecting the operation of the client VM executing on the corresponding client. In many modern applications of hardware virtualization, such as VDI, virtual machines are typically instantiated and removed dynamically on client systems based on VM images stored locally on the client or received from a VDI server computer. When the audited VM does not have a permanent audit agent, the VM image can be updated or otherwise changed at any time without affecting the audit software.

However, performing audit operations from outside the audited VM poses a difficult technical problem, commonly referred to in the art as "bridging the semantic gap." While software executing within the audited VM has access to a wealth of information relevant to the audit, software executing outside the respective VM typically only has access to the contents of physical memory and the current state of the audited VM's virtual processors. Consequently, unraveling the semantics of software within the VM from outside the respective VM may be impossible or, at best, require a significant computational effort. To help bridge the semantic gap, some embodiments place an audit agent within the audited VM. The audit agent can collect information from within the audited VM and communicate this information to components executing outside the audited VM.

It will be apparent to those skilled in the art that the above embodiments can be modified in many ways without departing from the scope of the invention. Therefore, the scope of the invention should be determined by the appended claims and their legal equivalents.

Claims (17)

1. A computer system comprising at least one hardware processor configured to execute a set of guest virtual machines (VMs) and further execute a VM auditing engine, the VM auditing engine being executed outside the set of guest VMs and configured to respond to receiving an audit request from a remote auditing server: An audit agent is inserted into the target VM of the group of customer VMs. The audit agent is configured to perform an audit of the target VM, which includes items selected from the group consisting of: generating a list of legal computer programs installed for execution on the target VM; and determining the amount of hardware resources currently used by the target VM. In response to the insertion of the audit agent, the target VM executes the audit agent; and In response to the target VM executing the audit agent, the audit agent is removed from the target VM; wherein inserting the audit agent includes: The target memory page is selected based on whether it contains part of the driver for the target VM's operating system. The driver loader is written to the target memory page, the driver loader is configured to load the audit driver into the target VM, and the audit driver is configured to perform the audit; and Configure the target VM to switch from executing the computer program to executing the driver loader when the computer program executing within the target VM issues a system call. 2. The computer system of claim 1, wherein the auditing further comprises determining the point in time when at least one computer program in the list of legitimate computer programs was installed. 3. The computer system of claim 1, wherein the auditing further comprises generating a second list of legitimate computer programs, wherein all members of the second list are currently loaded into the volatile memory of the target VM. 4. The computer system according to claim 1, wherein the hardware resources include the processing power of the at least one hardware processor. 5. The computer system according to claim 1, wherein the hardware resources include the non-volatile storage device of the computer system. 6. The computer system of claim 1, wherein the VM audit engine is further configured to: When preparing to insert the audit agent, the type of operating system currently running on the target VM is automatically detected; and In response, the audit agent is determined based on the type of the operating system. 7. The computer system according to claim 1, wherein: The audit request includes an indicator for the target VM; and The VM audit engine is configured to select the target VM from the group of customer VMs based on the audit request. 8. The computer system of claim 1, wherein the VM audit engine executes within an audit VM that is different from the target VM. 9. A method for executing a VM audit engine externally to a set of guest virtual machines (VMs) on a computer system using at least one hardware processor of the computer system, wherein executing the VM audit engine includes: In response to receiving an audit request from a remote server, an audit agent is inserted into the target VM of the group of customer VMs. The audit agent is configured to perform an audit of the target VM, which includes items selected from a group of: generating a list of legitimate computer programs installed for execution on the target VM; and determining the amount of hardware resources currently used by the target VM. In response to the insertion of the audit agent, the target VM executes the audit agent; and In response to the target VM executing the audit agent, the audit agent is removed from the target VM, wherein inserting the audit agent includes: The target memory page is selected based on whether it contains part of the driver for the target VM's operating system. The driver loader is written to the target memory page, the driver loader is configured to load the audit driver into the target VM, and the audit driver is configured to perform the audit; and Configure the target VM to switch from executing the computer program to executing the driver loader when the computer program executing within the target VM issues a system call. 10. The method of claim 9, wherein the auditing further comprises generating a second list of legitimate computer programs, wherein all members of the second list are currently loaded into the volatile memory of the target VM. 11. The method of claim 9, wherein the hardware resources include the processing power of the at least one hardware processor. 12. The method of claim 9, wherein the hardware resources include the non-volatile storage device of the computer system. 13. The method of claim 9, wherein the VM audit engine is executed within an audit VM that is different from the target VM. 14. The method of claim 9, further comprising: When preparing to insert the audit agent, the type of operating system currently running on the target VM is automatically detected; and In response, the audit agent is determined based on the type of the operating system. 15. The method according to claim 9, wherein: The audit request includes an indicator for the target VM; and The method further includes selecting the target VM from the group of customer VMs based on the audit request. 16. A non-transitory computer-readable medium storing instructions that, when executed by at least one hardware processor of a computer system, cause the computer system to form a virtual machine (VM) audit engine, the VM audit engine executing outside a set of guest VMs exposed on the computer system, wherein the VM audit engine is configured to respond to receiving an audit request from a remote audit server: An audit agent is inserted into the target VM of the group of customer VMs. The audit agent is configured to perform an audit of the target VM, which includes items selected from a group of: generating a list of legitimate computer programs installed for execution on the target VM; and determining the amount of hardware resources currently used by the target VM. In response to inserting the audit agent, the target VM executes the audit agent; and in response to the target VM executing the audit agent, the audit agent is removed from the target VM; wherein inserting the audit agent includes: The target memory page is selected based on whether it contains part of the driver for the target VM's operating system. The driver loader is written to the target memory page, the driver loader is configured to load the audit driver into the target VM, and the audit driver is configured to perform the audit; and Configure the target VM to switch from executing the computer program to executing the driver loader when the computer program executing within the target VM issues a system call. 17. The computer-readable medium of claim 16, wherein the instructions further cause at least one hardware processor of the computer system to form a hypervisor configured to expose the target VM on the computer system.