WO2024238711A1 - Systems and methods for waste management using a mobile application - Google Patents

Abstract

A system includes one or more data processors and a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations including: (a) receiving a scanned item from a mobile device; (b) placing the scanned item in a service queue for identification; (c) determining whether the scanned item is recyclable; and (d) based at least in part on the scanned item being recyclable, providing the mobile device with an indication of updated rewards.

Description

SYSTEMS AND METHODS FOR WASTE MANAGEMENT USING A MOBILE APPLICATION

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63/502,475, filed May 16, 2023, which is hereby incorporated by reference herein in its entirety.

FIELD OF THE INVENTION

[0002] The present invention relates generally to waste management systems, and more specifically, to mobile-based applications for identifying and classifying waste products.

BACKGROUND OF THE INVENTION

[0003] Waste materials from home, residences, businesses, stores, convenience stores, fast food chains, restaurants, retail locations, offices, schools, and commercial, institutional, and government environments or locations are discarded by individuals. These individuals can be homeowners, renters, occupants, workers, consumers, employees, students, etc. The individuals discard waste materials into waste receptacles provided onsite. Examples of waste receptacles include waste containers such as trash containers, trash receptacles, trash cans, garbage containers, garbage receptacles, garbage cans, etc. Waste receptacles are typically fitted with disposable plastic bags, which can be closed when filled, removed from the waste receptacles, and then disposed of in waste containers, usually located outside the buildings and/or adjacent to the parking lots of these locations, to facilitate removal by a waste disposal service.

[0004] Typically, the waste materials are not sorted onsite, and all different types of waste materials are discarded into the same waste receptacles. This lack of sorting of waste materials results in a wide variety of waste materials mixed together, including trash, garbage, paper products plastic products, food waste, and other waste items defining a general waste stream. The plastic bags filled with waste materials are thrown into a garbage can or dumpster and mixed with other loose waste materials or plastic bags containing other waste materials. The combined waste materials are transported in dumpsters or garbage trucks to landfills, typically without sorting the waste materials.

[0005] Currently, there exists a need for efficient processes, which require separating or sorting waste materials into targeted waste categories to allow facilitate recycling some of the waste materials. Furthermore, due to the large amount of labor involved with sorting, consumers of food and beverage products (e.g., individuals, end users, customers, students, employees, workers, contractors) are relied upon to commence the waste materials sorting process. For example, consumers are relied upon to sort waste food and beverage products by placing these items in marked waste collection containers (e.g., receptacles, bins, vessels). For example, consumers immediately place used paper cups or plastic bottles into separately marked waste collection containers to begin the process of effectively sorting particular types of waste. Costs associated with improper recycling can be around $75,000 per year per municipality or organization in the United States. The present disclosure provides solutions for improving the waste materials sorting process.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0007] The disclosure, and its advantages and drawings, will be better understood from the following description of representative implementations together with reference to the accompanying drawings. These drawings depict only representative implementations, and are therefore not to be considered as limitations on the scope of the various implementations or claims.

[0008] FIG. 1 provides an example first step in a workflow for recycling a product using a waste management mobile application, according to some implementations of the present disclosure.

[0009] FIG. 2 provides an example second step in the workflow for recycling a product using a waste management mobile application, according to some implementations of the present disclosure.

[0010] FIG. 3 proves an example third step in the workflow for recycling a product using a waste management mobile application, according to some implementations of the present disclosure.

[0011] FIG. 4 provides an example fourth step in the workflow for recycling a product using a waste management mobile application, according to some implementations of the present disclosure.

[0012] FIG. 5 provides contrastive pre-training, according to some implementations of the present disclosure.

[0013] FIG. 6 provides an example of filtering using vector analysis, according to some implementations of the present disclosure.

[0014] FIG. 7 shows an example of selective dimming based on cosine similarity.

[0015] FIG. 8 provides a screenshot of a leaderboard according to some implementations of the present disclosure.

[0016] FIG. 9 provides a screenshot of a wallet according to some implementations of the present disclosure.

[0017] FIG. 10 provides a screenshot of a user dashboard according to some implementations of the present disclosure.

[0018] FIG. 11 provides examples of tracking recycling according to some implementations of the present disclosure.

[0019] FIG. 12 provides example screen transitions according to some implementations of the present disclosure.

[0020] FIG. 13 provides screen transitions that extend those of FIG. 12.

[0021] FIG. 14 is a system for waste management, according to some implementations of the present disclosure.

[0022] FIG. 15 is a flow diagram showing a process for waste management, according to some implementations of the present disclosure.

[0023] FIG. 16 provides an example screenshot showing results of textual identification, according to some implementations of the present disclosure.

[0024] FIG. 17 provides an example screenshot of composting recommendation, according to some implementations of the present disclosure.

[0025] FIG. 18 provides an example screenshot of a marketplace for spending rewards, according to some implementations of the present disclosure.

[0026] FIG. 19 provides an example screenshot having multiple water bottles, according to some implementations of the present disclosure.

[0027] FIG. 20 provides an example screenshot showing results of identifying some of the water bottles in FIG. 19.

[0028] FIG. 21 provides an example architecture for contrastive captioning, according to some implementations of the present disclosure.

[0029] FIG. 22 provides a training protocol for an artificial intelligence model, according to some implementations of the present disclosure. [0030] FIG. 23 provides a matching inference protocol for identifying items, according to some implementations of the present disclosure.

[0031] FIG. 24 provides a captioning inference protocol for identifying items, according to some implementations of the present disclosure.

DETAILED DESCRIPTION

[0032] Implementations of the present disclosure provide a waste management system for facilitating sorting of waste materials. The waste management system can be provided on a mobile application accessible to a user and allow for scanning (i.e., imaging, capturing or photographing) of recyclable, compostable, and other waste materials. The waste management system can be used to identify and store waste materials that the user has scanned under a single profile of the user. In some implementations, the user is able to earn points for recycling. The points can be used for purchasing items through vendors. The waste management system provides the user valuable information and improves user confidence when placing items in garbage bins, recycle bins, etc.

[0033] Various implementations are described with reference to the attached figures, where like reference numerals are used throughout the figures to designate similar or equivalent elements. The figures are not necessarily drawn to scale and are provided merely to illustrate aspects and features of the present disclosure. Numerous specific details, relationships, and methods are set forth to provide a full understanding of certain aspects and features of the present disclosure, although one having ordinary skill in the relevant art will recognize that these aspects and features can be practiced without one or more of the specific details, with other relationships, or with other methods. In some instances, well-known structures or operations are not shown in detail for illustrative purposes. The various implementations disclosed herein are not necessarily limited by the illustrated ordering of acts or events, as some acts may occur in different orders and/or concurrently with other acts or events. Furthermore, not all illustrated acts or events are necessarily required to implement certain aspects and features of the present disclosure.

[0034] The following are definitions of terms used in this disclosure that relate in general to waste management systems.

[0035] An organization is a university, company or municipality that has a recycling program.

[0036] A gift/reward organization is a vendor which a user can redeem their points with for gift cards and coupons

[0037] A bin is a single receptacle which accepts specific items.

[0038] A recycling station is a collection of one or more bins.

[0039] A gift offer/reward is an offer made by gift organizations that users can redeem their points for.

[0040] A user is an individual who has signed up on the waste management mobile application. In some implementations, each user belongs to an organization. In some implementations, their organization is assigned based on their sign up email’s domain address (e.g., “@gatech.edu”).

[0041] A wallet contains any gift offers a user has yet to redeem.

[0042] A leaderboard shows the user’s standing in total number of points relative to the other users within their organization.

[0043] A level of a user is determined by the number of items the user has recycled. The user’s current level can be displayed on the user’s home page in the waste management mobile application.

[0044] A material is what an item is made of (e.g., plastic, metal, paper, etc.).

[0045] An item is a specific object made of a material (e.g., water bottle, soda can, grocery bag, cotton, textile materials, etc.).

[0046] A deposit is an image containing an item. The image is uploaded by a user.

[0047] A validator is a contractor or employee who is responsible for reviewing deposits and improving their labels.

[0048] A recycling coordinator is an organization employee who is responsible for the recycling program. The recycling coordinator will have access to an organization specific administrative dashboard where they can control the list of acceptable items and view aggregate details of their respective users’ recycling behavior.

[0049] FIGS. 1-4 provide example steps in a workflow for recycling a product using a waste management mobile application, according to some implementations of the present disclosure. In FIG. 1, a user takes a picture of an item to be recycled. For example, a paper cup is provided in the camera view. In FIG. 2, the waste management mobile application detects which items or materials in the picture are recyclable and which are not. In some implementations, an icon representing “recyclable” and an icon representing “not recyclable” can be provide. In some implementations, color codes can be used, for example, the color green used for recyclable and the color red used for not recyclable. In some implementations, an alert is provided to the user to remove all unrecyclable items from the bin (or the plastic bag).

[0050] In FIG. 3, the user is rewarded points for recycling. The waste management mobile application can display the number of items the user has recycled and can provide a number of items remaining before the user reaches a recycle goal. In FIG. 4, the waste management mobile application can provide the user the opportunity to redeem points for discounts on products and services. Rewards can include gift cards, coupons/discounts, donations to charities or other users, cashback opportunities, etc. For example, FIG. 18 provides a screenshot of a marketplace for spending rewards. As provided in FIG. 18, the user can spend reward balance of 550 on “Burger King” virtual gift card, “adidas” virtual gift card, and “Booster Juice” virtual gift card. “Pizza Hut” virtual gift card is locked because the reward balance of 550 does not meet the minimum of 1000 required for the “Pizza Hut” virtual gift card.

FEATURES OF THE MOBILE APPLICATION

Localization feature

[0051] Waste management mobile applications according to some implementations of the present disclosure provide users with personal profiles (i.e., user profiles). For users to benefit from personalized customization of their user profiles, in some implementations, the user provides information that is stored in the user profile. For example, the user provides an email address for registration. The user’s email address can provide information associated with organization affiliation. For example, the waste management mobile application can analyze the user’s email and determine an affiliated organization based on the email. The domain of the email can be used to determine organization. For example, an email using “@gatech.edu” can associate the user as being affiliated with Georgia Tech.

[0052] Determining the affiliated organization can provide further customizations for the user. For example, the knowledge of the organization can be used to determine how the mobile application appears on the user’s mobile device. In an example, the affiliated organization can determine what type of recycling programs are available. That is, what kind of items can be recycled at the specific organization, what kinds of rewards or discounts programs are available, etc. In an example, the recycling programs available at the organization are stored in a database, and the recycling programs can be retrieved for operating the mobile application.

[0053] In addition to information regarding the recycling programs available, the affiliated organization can also determine aesthetic components of the mobile application. For example, co-branding that allows the mobile application to change color schemes based on the organization is available. For example, school colors or insignia can be displayed on the mobile application. Georgia State affiliation can provide blue colors while Georgia Tech affiliation can provide yellow colors.

[0054] In some implementations, subgroupings are possible. Municipalities (e.g., city of Atlanta) can also be attached to the user’s profile. Although city is used as an example, counties and other groupings are envisioned. Recycling programs can differ at any of these granular levels. That is, an organization can provide recycling programs independent of the municipality. The recycling programs granted by the organization may or may not be more stringent than those granted by the municipality. Therefore, identifying locality information for the user (i.e., municipality and/or organization) allows the mobile application to determine rules and recycling program.

[0055] In some implementations, the user’s email can be used to modify some settings while the user’s location can be used to modify other settings. The setting modified by email or location can be the same or different. For example, special rates (e.g., reward rates) can be generated for students if the students use a “.edu” email address. The students can have the option of choosing whether the mobile application should default to their municipality’s recycling program or whether to default to their school’s recycling program. Although school is used here as an example, this applies to organization vs. municipality as well. In some implementations, the mobile application switches between school’s and municipality’s recycling programs based on geolocation so that depending on where the user is located, the mobile application automatically refreshes to match recycling options available at the location.

[0056] In some implementations, email address is not used for identifying municipalities or organizations. The user can be prompted for information. For example, the user can be prompted “Are you in Atlanta?”, “Are you affiliated with Georgia Tech?”, etc. The user can be given the option to join a municipality or a specific organization. Geolocation information can be obtained from the mobile device of the user to tailor the prompts to the user. For example, the waste management system can include geotagged bins at different locations. The user’s mobile device can sense proximity to these geotagged bins to provide approximate locations for tailoring prompts to the user. In some implementations, GPS of the mobile device is used. [0057] In some implementations, co-branding changes based on municipality and/or organization. Recycling program can change based on municipality and/or organization. Different organizations can operate locally, thus specific partnerships for rewards, etc., can change based on municipality and/or organization. Data schema for recycling program can change between organizations and/or municipalities. For example, an item can be recyclable based on Georgia State recycling program and not just general recyclability. The user can be provided directions by the mobile application for where to recycle and how to recycle. The mobile application can also indicate that the item is generally recyclable, but the specific organization or location is not able to recycle the item. For example, Georgia State University might not recycle the specific item but Georgia Tech can.

[0058] In some implementations, blogs, pictures, news, color theme, rules associated with what is recyclable will change based on municipality and/or organization. White label products/items will also change. Reward can also be associated with the school/ organizati on/1 ocati on .

Capture feature

[0059] As shown in FIG. 1, a user can take a photo of the item to be recycled using the mobile application. The capture feature of the mobile application allows making a deposit using a camera (e.g., a camera of the mobile device of the user).

[0060] In some implementations, the user clicks a center button in the mobile application to take the photo. The photo taken will be analyzed to identify one or more items in the photo. In some cases, an artificial intelligence (Al) model looks at the database for identifying the items in the photo. The mobile application does not just predict, in general, what the item is. The mobile application looks at the database relevant to the organization and bin. The Al model ranks item similarity relative to that specific bin. For example, if Georgia State does not accept pizza boxes but Georgia Tech does, then the mobile application can show stations available for recycling the pizza box. That is, a random pizza box with nowhere to recycle on Georgia State campus can be referred to one or more bins on Georgia Tech campus. In some implementations, the mobile application provides directions for reaching at least one of the bins on Georgia Tech campus for recycling the pizza box.

[0061] In some implementations, the waste management system can deal with recycling items that are not clear or missing in the photo. For example, the photo only shows a partial view of the item or shows a perspective of the item that is not preferable. For example, the bottom of a bottle is shown rather than the entire shape of the bottle that would facilitate identifying the item as a bottle. The waste management system can analyze the photo to identify items for recycling. In an example, the mobile application can prompt the user to rotate/move items, take a new photo, and/or do nothing if the item is recognizable.

[0062] In some implementations, with enough data and resources, the Al model can be trained to handle situations of partial and missing items. When using pre-trained Al models and without fine-tuning, there are several options for dealing with partial and/or missing items.

[0063] For example, most Al vision software today leverages convolutional neural network (CNN) models. For partial items, a vision transformation (ViT) model which uses selfattention layers instead of convolutional filters can be leveraged. The main difference between convolutional filters and self-attention layers is that the new value of a pixel depends on every other pixel of the image. As opposed to convolution layers whose receptive field is the K-squared neighborhood grid, the self-attention's receptive field is always the full image. Therefore, as an example, if the pixels of a water bottle make up only 15% of the image and is heavily skewed to the left, that image is still closer to the “water bottle” item embeddings in the database than it is the “soda bottle” item embeddings item and the ranking predictions are still relevant to the user.

[0064] In the case of a missing item, the ranking predictions can inherently be irrelevant. As the item embedding latent space is enriched in a database, in some implementations, users can be informed that the image does not produce an item embedding that is similar enough to any of the other deposits that have been labeled in the database. An “unknown item detected” warning can be returned to the user’s device, and the user can be asked to retake the image and/or submit a custom description.

[0065] Although the ViT architecture is described herein, CNN models can also be used in certain situations. The ViT architecture can be more robust than CNN models for augmented images or images with noise. ViT can perform better compared to CNN due to the selfattention mechanism because it makes the overall image information accessible from the highest to the lowest layers. On the other hand, CNN models can generalize better with smaller datasets and can obtain better accuracy than ViT. But ViT can learn information better with fewer images because the images are divided into small patches, thus creating a greater diversity of relationships between the images.

Review of Captured Items

[0066] In some implementations, captured items are marked as “Pending” in the mobile application. Review process can change the captured items from “Pending” to “recyclable” vs “not recyclable” based on specific recycle program as described above, specific types of material (e.g., plastic, metal, paper, etc.), specific type of item (e.g., water bottle, chip bag, etc.), etc. While in the “pending” state, the captured items can be placed in a service queue before determining to change the status of the captured items to “recyclable” vs “not recyclable”.

[0067] In some implementations, the review process is partially manual using an administrative dashboard. For example, a generalized Al model can be used with labeled data obtained from a validator. The validator can mark and classify images recyclable, unrecyclable, plastic, metal, paper, water bottle, cardboard box, etc. In an example, OpenAI model being used in a different way. Al at high level takes a picture and generates a caption for that picture. Example of captions include “a glass of water sitting on a desk.” The Al takes in the image and it generates numeric representation of the meaning of the image. The database contains text description of all the items, and the numeric representation of the text description of the items can be stored. The numeric representation of the image is then compared to the numeric representation of the text description of all the items. Then a ranking is performed.

[0068] In some implementations, contamination analysis of objects can be trained in a similar manner. Human validators can provide good and band examples (e.g., a dirty bottle vs. a clean bottle). Recyclable item can prompt giving a warning to clean the bottle. The waste management system can validate that the item is cleaned.

[0069] Items are constantly being added and refined as more images come into the mobile application. For the text database, a first step may include uploading the university-specific accepted and rejected items list. Over time, users, validators, and recycling coordinators enter in new items or refine existing labels, and after validator or recycling coordinator approval, those items are then added to the program. For example, a user may add a picture of a “water bottle”. The validator or administrator can then change the label to “Dasani water bottle” and add that label to the university’s list of acceptable items. Once it is added, a user who submits an image of a Dasani water bottle will most likely see the new “Dasani water bottle” label as the first prediction result.

[0070] Location specific items: an organization's recycling coordinator is responsible for the items that are considered acceptable or not to their program. The validators are responsible for approving whether or not images match existing items, and relabeling those items to have more detailed information.

[0071] In some implementations, vector representation is used for the image, and vector embeddings are used for the text. Cosine similarity can be used for ranking. FIG. 5 provides a process for contrastive pre-training. For example, a contrastive language-image pretraining (CLIP) model which learns visual concepts from natural language supervision can be used. The CLIP model produces 1024 length embeddings for both text and images. The latent space is shared between the text and vision model, making it possible to easily perform cosine similarity between text and image vectors. Although CLIP model is described, other contrastive learning approaches can be used in other implementations. Contrastive learning works on the principle of juxtaposing samples from a dataset and push or pull representations in the embedding space based on whether the samples belong to the same distribution (i.e., the same class in classification tasks or the same object in object detection/recognition tasks) or different distributions respectively. Thereby, image to image mapping, image to text matching, etc., can be used for item identification. Thus, the framework of using juxtaposition to train using both negative (opposite) samples against positive (similar) samples can be applied to different modalities (image vs. text).

[0072] For multi-object detection, many image patches can be created at different levels of resolutions, and predictions can be made on every patch. Cosine similarity can be performed between each patch and every possible item. Normalization, filtering, and region proposal can be applied to then identify multiple items in the image. A similar process is demonstrated in FIG. 6, but instead of filtering on pixel colors as is done in FIG. 6, implementations of the present disclosure filter on the cosine similarity.

[0073] In some implementations, automated review can be performed, thus avoiding a queuing system. The automated system can include anti-fraud provisions. For example, beacons can be placed on bins that verify if a user was in a certain place at a given moment of the day. By knowing the bin location and the user’s distance from that bin, the waste management system can assume that the user has deposited items into the bin.

[0074] In some implementations, the waste management system along with the mobile application rate limits the number of items to be deposited. For example, the mobile application can prevent depositing more than 50 items within a 30 second span. This can prevent spamming pictures. In some implementations, the rate limit is based on people’s habits. For example, some individuals can be allowed to recycle in batches and others recycle one at a time. Images can be timestamped to indicate when the images were collected, and in some cases, images are required to be uploaded at the moment they are captured.

[0075] In some implementations, validators can be onboarded so that the individual scanning the product is different from the user. For example, validators can be dorm room managers that encourage recycling and help students scan items being recycled.

[0076] In some implementations, every captured image is compared to prior images from the specific user and/or other users to determine whether the captured image is unique. The specific scene surrounding an item can be analyzed as well.

[0077] In some implementations, multiple object detection is provided. Multiple object detection involves looking for various items in the same photo. For example, FIG. 19 provides an example screenshot having multiple water bottles. FIG. 20 provides an example screenshot showing results of identifying of the water bottles in FIG. 19. Each of the bottles is identified as a subsection in the overall photo and tagged with a number. In this example, there are 15 water bottles identified as “water bottle jug” with each of the water bottles having a number between 1 and 15, inclusive. In some implementations, the user can delete misidentified the mobile application can provide bounding boxes or a list of items in the scene, and the user can select which items are to be recycled.

[0078] For example, a user can take a photo that includes all items the user intends to recycle. The mobile device orientation for multiple object can be landscape mode vs. portrait mode for single object. That is, when the user wants to indicate to the mobile application that more than one item is to be detected in a photo, then the user can take the photo in landscape orientation. For example, in FIG. 19 the example screenshot has the multiple water bottles in landscape orientation indicating to the mobile application that multiple objects are to be detected/identified. In some implementations, the mobile application can nudge the user to switch to landscape mode if the user is uploading more than one object. In some implementations, the mobile application can nudge the user to give a number of items beforehand and based on the number being above an item threshold, suggest the user use landscape mode.

[0079] In some implementations, instead of using traditional bounding boxes, images can be divided into patches, and based on an attention map, certain patches can be dimmed. The attention map is an output of the transformer architecture. The attention map is averaged, normalized, and then recursed upon to essentially perform a backpropagation, resulting in a final attention map that has the weighted sum of attention of all upstream maps for every pixel in the image. This weighted sum for every pixel is then used to set the dim level on each pixel in the image, thus providing an importance overlay on top of the image. Regions can be proposed around these patches which could be used to produce bounding boxes. In some implementations, this type of display can be more memorable to users than traditional bounding box visualization. FIG. 7 shows an example of selective dimming based on cosine similarity.

[0080] In some implementations, the waste management system can use video for validation instead of images. Users can record themselves throwing items into bins. In some implementations, sensors are placed on the bins or specific locations can be designated as recycling spots. For example, supermarkets can be designated as recycling spots. Supermarkets can be good for recycling plastic bottles and aluminum cans, even though the waste management system can detect any material. Supermarkets are limited to recycling at the specific location so not scalable. Supermarkets have no identity associated with who is making deposit, so cannot determine patterns associated with depositors. Supermarkets know what you buy and what you throw away so marrying the waste management system with the supermarket’s databases can provide additional insights.

[0081] In some implementations, depth and perspective information can be gleaned from captured images for fraud detection. In the case where the user takes a picture of a screen or a picture of a photograph to try to fool the waste management system, depth and perspective information can be used to reject the captured images by the user. The user’s mobile device can include at least two cameras separated by a camera distance. For example, the at least two cameras can be included on a backside of the user’s mobile device such that each respective camera is able to capture a respective camera image. The respective camera images can be used along with the distance between the cameras to determine a depth associated with the image. A larger displacement will be observed between the camera images for objects closer to the cameras compared to objects farther away from the cameras. In some implementations, the mobile application can suggest that photographs of items be taken at a certain distance where a larger displacement can be measured between the camera images in order to reduce the likelihood of a fraudulent image being analyzed.

[0082] Depth information, when available, can be used to estimate volume of an item or waste material captured. In some implementations, weight of the item can be determined from the estimated volume based on material density information stored in the database.

[0083] Depth information can be used for fraud detection in other scenarios. For example, when a depth of an object associated with an image scene is below a depth threshold, then an alarm is raised to indicate that the perceived object is flat. For example, the depth threshold can be 2, 5 or 10 centimeters. In some implementations, detecting object depth of under 3 meters can be achieved. So, for fraud detection, certain depth thresholds can be assigned to certain classes of items.

[0084] In some implementations, companies or organizations can determine how much of their products get recycled. Shopping stores (e.g., Amazon) can tie purchases to recyclables. Online stores can determine how many shipping boxes should be recycled by their users. Beverage companies can determine how many bottles should be and how many bottles have been recycled. Some companies can give rewards on receipt. Recyclability of items purchased can be determined at time of purchase. The mobile application can use optical character recognition (OCR) to read what is in the receipt image through text. Based on process of elimination, try to figure out what the user throws away. User buys a bunch of stuff and the mobile application keeps track and checks off items that have been disposed of on the list of items purchased. In some implementations, if an item can be linked back to a purchase, points can be awarded differently. The mobile application can proactively provide user with information on what to put in the trash vs. what not to trash.

[0085] In some implementations, points can be based on user input, for example, points can be based on material type, picture, description, etc. The market value of materials changes, so rewards can be tied to market value. This ties the rewards or point system to the market. In an example, cardboard can be more valuable can plastic, so higher rewards are provided for recycled cardboard over recycled plastic. In some implementations, the location where the user makes the recycling action can affect the points earned (e.g., recycling a burger king cup at a burger king location).

Gamification

[0086] In some implementations, a game can be set up where items should be recycled by a specific time. For example, the system can be set up with a challenge where students recycle 'X' amount of materials on campus or in their locality by a specific date. The users (e.g., the students) can add friends to these challenges and can win a collective prize or donate winnings to a charity. Budgets for certain amount of points can be set up.

[0087] In some implementations, the waste management system can adjust rewards point to consider environmental impact score. The environmental impact score can be a carbon index or an environmental footprint index. FIG. 11 provides an example of a carbon footprint calculator that includes index contributions from recycling practice, use of renewable energy (e.g., solar power) vs. non-renewable energy, carpooling, electric vehicle use vs. gas or petroleum vehicle use, public transit use, using eco-friendly vs. unrecyclable products, etc. The environmental footprint index can be determined to be a weighted contribution from some or all of the aforementioned factors. The environmental footprint index can be expressed in a separate leaderboard. The environmental footprint index can be further based on the individual purchasing environmentally conscious brands (e.g., a company that makes clothing out of recycled products). Environmentally conscious brands can exhibit a higher weight or can have associated premium points compared to brands that are not environmentally conscious. In some implementations, use of ridesharing services (e.g., Uber) can reduce the environmental footprint index relative to the cost of using carpooling services or even public transportation (e.g., the bus or train). In some implementations, the environmental footprint index can tie in with when the individual donates to charities in-app (e.g., donating to a Plant a tree charity).

[0088] In some implementations, the mobile application can provide for avatars or characters for leveling up. Making it more user friendly and wanting the user to use the mobile application for recycling purposes. Competition events can be provided (service and surface these events through the app, grant scholarships, special recognition). During automatic validation, take a picture and confidently say water bottle and no contaminant and can have an explosion of confetti or animation in the app.

[0089] In some implementations, the mobile application can provide a leaderboard for different competition, different organizations, etc. FIG. 8 illustrates a screenshot of an example leaderboard. Leaderboard (e.g., FIG. 8) inside an organization. State leaderboard and/or global leaderboard can also be provided. In some implementations, provide users with information on what has been learned via a quiz, data to show universities that they are improving on recycling goals, surveys on where users recycle, etc.

[0090] Implementations of the present disclosure can encourage municipalities to provide tax credits for companies based on recycling merely by having users registered on the system. The mobile application allows communication with users. Organization can send notifications to users about programs, ways to get reward points, etc.

General Monitoring of Waste

[0091] In some implementations, the mobile application can be used to monitor not just recycling, but can be expanded to other waste management services. Rewards can be adjusted based on multipliers. Firstly, the value of an item should be highly correlated with its value in terms of recycling revenue. Larger items should have higher values, and different materials should have higher values than others (e.g., glass vs. plastic). Secondly, certain materials recovery facilities (MRFs) only accept certain materials. Therefore, depending on which bin you are recycling at, the item may not be awarding points because it won’t end up at an MRF that will process it. This relationship is meant to be managed by the recycling coordinators through our admin dashboard as well as our validators when adding new acceptable items to the existing organization’s list.

[0092] FIGS. 11 and 12 provide example app screen transitions according to some implementations of the present disclosure. After an individual signs up or logs in, the individual can be presented a dashboard screen. At the dashboard screen, the individual can be shown points, items recycled, goals, etc. From the dashboard screen, the individual can transition to the rewards page to view a history or reward activity. From the dashboard screen, the individual can also transition to the recycling lobby that allows scanning of items as described above, according to some aspects of the present disclosure. Turning to FIG. 12, from the recycling lobby, the individual can access and transition to the community page (FIG. 13) to view the leader board and recycling activities from other users. For example, the individual can be provided an update on a number of bottles a user Joe recycled, can be provided an update on who the top recycler is for a period of time (e.g., for the week, the month, etc.), can be provided trends in the specific area for the individual, can be provided updates on movements on the leaderboard, etc.

[0093] Some implementations of the present disclosure provide a tracking system to monitor a life cycle of a recycled item from point of pickup to upcycling and final conversion. Users can verify the origin of the purchased product and ensure that once finished with the product, the packaging, or items meant for recycling, are truly upcycled due to the entire history of product being on record (via blockchain or other mechanism).

[0094] Users are not the only ones that can track products. Some implementations of the present disclosure enabling transparency for institutions by providing insights into the logistics process of recycling. For example, institutions can track the logistics process from curbside pickup to sorting center to processing at material recovery facilities to upcycling facilities and finally back to reintroduction in the market.

[0095] FIG. 14 is a system 1400 for waste management, according to some implementations of the present disclosure. The system 1400 includes one or more client devices 1404, a server 1402, and a database 1406. Each of these components can be realized by one or more computer devices and/or networked computer devices. The one or more computer devices include at least one processor with at least one non-transitory computer readable medium. In some implementations, the one or more client devices 104 is a smartphone or tablet computer with a camera for capturing images. In some implementations, the server 102 stores an Al algorithm for analyzing the captured images for managing specific waste materials or items identified in the captured images. In some implementations, the database 106 includes files, images, computational models, settings and configurations, etc., used by the server 102 for analyzing the captured images and generating results provided to the one or more client devices 104.

[0096] In some implementations, the one or more client devices 104 are two devices, with a first client device being used by an administrator to configure settings associated with waste management programs and a second client device being used by a user for capturing images of items that will undergo waste management. In some implementations, the first client device is a laptop computer, a desktop computer, a tablet, a smartphone, etc. In some implementations, the second client device is a smartphone or a tablet computer with at least one camera. In some implementations, the second client device includes two cameras for depth estimation.

[0097] The server 102 and the second client device can, in cooperation, perform localization as described above. For example, the second client device can provide location identifying information (e.g., email address, GPS location information, city, proximity to geotagged bins, etc.) to the server 102 such that the second client device can display customized views to the user of the second client device.

[0098] In some implementations, the server 102 includes Al models for identifying items within captured or scanned images as described above. For example, parameters for the CLIP model can be stored in the database 1406, such that the server 102 retrieves those parameters for determining cosine similarities for identifying representative text that represents items within the captured or scanned images. In some implementations, the server 102 can perform the ranking of similar items relative to specific bins as described above. The server 102 can provide alternative stations to the client device(s) 1404, such that, for example, the second client device can use a mapping feature to reach the alternative stations. [0099] In some implementations, the client device(s) 1404 includes a preprocessing operation as discussed above, where an image capturing only a partial view of an item is not acceptable and the client device(s) 1404 displays a prompt to retake the image of the item. In some implementations, the client device(s) 1404 can include preprocessing operations for patches and selective dimming as discussed above.

[00100] The database 1406 can store recycling programs information for different granular levels as described above. For example, a computing system used by an administrator (e.g., one of the client devices 1404) can be used to configure the recycling program for an organization, and these recycling program settings can be stored in the database 1406 for use by the server 1402. Similarly, composting settings, textile settings, and other waste management settings can be stored in the database 1406. The database 1406 can store profile settings for users.

[00101] FIG. 15 is a flow diagram showing a process 1500 for waste management, according to some implementations of the present disclosure. The process 1500 is performed by the system 1400, and in some cases, more specifically, by the server 1402. At step 1502, the server 1402 receives an image from a mobile device (e.g., the client device(s) 1404). The image includes at least one item (i.e., at least one scanned item). In some implementations, the image includes metadata that can be used by the server 1402 to identify a location where the image was captured. In some implementations, the image includes metadata that can be used to identify a displacement between two camera images for obtaining depth information from the image. In some implementations, the server 1402 receives, along with the image, data identifying a user associated with the mobile device (e.g., email, profile, etc.).

[00102] At step 1504, the server 1402 places the at least one item in a service queue for identification. When in the service queue, a status associated with the at least one item can be set to “pending” indicating that the item is currently being identified. In some implementations, the “pending” state indicates that a bucket associated with the item is currently being determined. The bucket can include whether the item is recyclable, not recyclable, compostable, textile, etc. In some implementations, the bucket can further include materials like: a. plastic (a synthetic material made from a wide range of organic polymers such as polyethylene, PVC, nylon, etc., that can be molded into shape while soft and then set into a rigid or slightly elastic form); b. cardboard (pasteboard or stiff paper); c. glass (a hard, brittle substance, typically transparent or translucent, made by fusing sand with soda, lime, and sometimes other ingredients and cooling rapidly. It is used to make windows, drinking containers, and other articles); d. metal (a solid material that is typically hard, shiny, malleable, fusible, and ductile, with good electrical and thermal conductivity (e.g., iron, gold, silver, copper, and aluminum, and alloys such as brass and steel)); e. paper (material manufactured in thin sheets from the pulp of wood or other fibrous substances, used for writing, drawing, or printing on, or as wrapping material); f. Styrofoam (a kind of expanded polystyrene); g. mixed (a multi-material object in which the materials are inseparable (e.g. potato chip bags); h. plastic film (a thin continuous polymeric material); i. battery (a device that produces electrical energy from chemical energy); j . organic (items which are generally compostable); k. clothing (fabric items which may be donated); and/or l. paperboard (a thick paper -based material. While there is no rigid differentiation between paper and paperboard, paperboard is generally thicker).

[00103] At step 1506, the server 1402 determines a waste management classifier associated with the item. For example, the waste management classifier can include a textual identification of the item along with the bucket associated with the item. For example, the textual identification can be “plastic bottle” and the bucket can be “recyclable.” In an example, FIG. 16 provides a screenshot of a mobile application running on the client device 1404 showing results of textual identification of a banana peel. The mobile application provides suggestions “banana peel”, “orange peel”, “food scraps”, and “apple core”. The client device 1404 can select “banana peel” to correspond to the picture.

[00104] At step 1508, the server 1402 provides the mobile device with an indication based on the waste management classifier. For example, if the waste management classifier is “recyclable” and the server 1502 confirms that the item has been recycled, then the server 1502 provides the mobile device with an indication of updated rewards as described above (e.g., updated rewards associated with recycling). In another example, if the waste management classifier is “compostable” and the server 1502 confirms that the item has been placed in a bin, then the server 1502 provides the mobile device with an indication of updated rewards associated with composting. For example, FIG. 17 provides a screenshot of composing recommendation for a banana peel that has been classified as “compostable”. In FIG. 17, 5 points of reward can be gained from placing the organic banana peel in a composting bin located in the identified station (e.g., Burgess Hall pictured).

[00105] FIG. 21 provides an example architecture for contrastive captioning, according to some implementations of the present disclosure. Contrastive captioning is provided herein an example for matching items to item descriptions. A dataset can be built. Images in the dataset can be labeled as follows: item (text data), material (category), brand (text data), dirty (true or false), fraud (true or false), multi-object (true or false), no object (true or false). In some implementations, to narrow evaluation space, a list of 100 most common items and materials are derived set aside as a primary set for evaluation.

[00106] Contrastive captioning can be used to obtain labels for items during item identification process as described above. In FIG. 21, an image is received by a vision encoder with potential labels received at a text encoder. FIG. 5 provides an example of a CLIP architecture repeated here in FIG. 21. The contrastive loss is obtained, and the captioning loss is also obtained to determine the joint loss. Training can be performed to reduce the joint loss, or sometimes users can provide new captions for items not yet included in the database of items.

[00107] FIG. 22 provides a training protocol for an artificial intelligence model (e.g., a contrastive captioning model), according to some implementations of the present disclosure. In a first step, an input batch is created. A sample of an even group of deposit images, their respective item IDs, and product captions are created. In a second step, vision embeddings are created. All images from the first step are fed through a pretrained contrastive captioning vision encoder and then split into two mini batches A and B.

[00108] In a third step, a Boolean mask is built. Similar to the second step, items are split into mini batches A and B and the Boolean mask is created based on the element-wise equality between items in batches A and B. In a fourth step, a cosine similarity matrix is obtained between batches A and B. In a fifth step, image captions are created via a cross attention connection between the vision encoder and the text decoder.

[00109] In a sixth step, the caption loss (CapL) between the output of the text decoder and the product captions for each sample is calculated. This value is added to the contrastive loss (ConL) between the batches A and B to produce the joint loss. Both the vision encoder and the text encoder are fine tuned on the objective of reducing the joint loss.

[00110] FIG. 23 provides a matching inference protocol for identifying items, according to some implementations of the present disclosure. In a first step, embeddings are added to a deposits table. For all existing deposits, the vision encoder is run on their images to generate or create a new embedding column containing the encoded image array.

[00111] In a second step, new images are downsized and masked for personally identifiable information (PII) and stored in a database (e.g., the database 1406). In some implementations, these images can be added to a file cache. In a third step, a user (e.g., using the client device 1404) submits an image to a server (e.g., the server 1402). In a fourth step, the server matches the encoding of the submitted image to 6 most similar deposit images using, for example, maximum inner product search (MIPS) and returns these images to the user.

[00112] In a fifth step, the user picks the right item from the group of images. Thus, the user can help with verifying identification of items.

[00113] FIG. 24 provides a captioning inference protocol for identifying items, according to some implementations of the present disclosure. In some implementations, instead of MIPS, users can pre-select a type of deposit. In some implementations, prompting and MIPS retrieval pathways are weighted and compared to determine which provides a more trusted text label.

ALTERNATIVE IMPLEMENTATIONS

[00114] Implementation 1. A system, comprising: one or more data processors; and a non- transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations including: receiving a scanned item from a mobile device; placing the scanned item in a service queue for identification; determining whether the scanned item is recyclable; and based at least in part on the scanned item being recyclable, providing the mobile device with an indication of updated rewards.

[00115] Implementation 2. The system of Implementation 1, wherein the one or more data processors are caused to perform the operations including: receiving the scanned item from a scene capture that includes multiple items.

[00116] Implementation 3. The system of Implementation 2, wherein the scene capture is captured in a landscape mode of the mobile device. [00117] Implementation 4. The system of Implementation 2 or Implementation 3, wherein the one or more data processors are caused to perform the operations including: causing the mobile device to nudge a capture of the scene in landscape mode.

[00118] Implementation 5. The system of any one of Implementations 2 to 4, wherein the determining whether the scanned item is recyclable includes using an artificial intelligence model to classify the scanned item as recyclable or unrecyclable.

[00119] Implementation 6. The system of any one of Implementations 2 to 5, wherein the determining whether scanned item is recyclable includes using an artificial intelligence model to identify a material associated with the scanned item.

[00120] Implementation 7. The system of any one of Implementations 2 to 6, wherein the determining whether the scanned item is recyclable includes determining whether the scanned item is contaminated.

[00121] Implementation 8. The system of any one of Implementations 1 to 7, wherein an amount of the updated rewards is based on a time provided recycling the scanned item.

[00122] Implementation 9. The system of any one of Implementations 1 to 7, wherein the amount of updated rewards is based on a position on a leaderboard.

[00123] Implementation 10. The system of any one of Implementations 1 to 9, wherein the updated rewards is based on a game.

[00124] Implementation 11. The system of any one of Implementations 1 to 10, wherein the one or more data processors are caused to perform the operations including displaying on a screen of the mobile device information regarding the scanned item.

[00125] Implementation 12. The system of Implementation 11, wherein the information is displayed as a quiz.

[00126] Implementation 13. The system of any one of Implementations 1 to 12, wherein the one or more data processors are caused to perform the operations including determining whether a scanned limit threshold is reached and determining that the updated rewards is zero when the scanned limit threshold is reached.

[00127] Implementation 14. The system of Implementation 13, wherein the scanned limit threshold is 50 items.

[00128] Implementation 15. The system of any one of Implementations 1 to 14, wherein the one or more data processors are caused to perform the operations including determining whether a scanned limit rate is reached and determining that the updated rewards is zero when the scanned limit rate is reached. [00129] Implementation 16. The system of Implementation 15, wherein the scanned limit rate is 50 items in 30 seconds.

[00130] Implementation 17. The system of any one of Implementations 1 to 16, wherein the one or more data processors are caused to perform the operations including determining an environmental footprint index based at least in part on metrics associated with a recycling practice, a use of renewable energy, a use of rideshare services, a use of electric vehicles, a use of gas or petroleum vehicles, a use of public transit, a use of eco-friendly products, a use of unrecyclable products, a support of environmentally conscious brands, donating to environmental charities, or any combination thereof.

[00131] Implementation 18. The system of Implementation 17, wherein the metrics are combined as a weighted combination to determine the environmental footprint index.

[00132] Implementation 19. The system of Implementation 17 or Implementation 18, wherein the one or more data processors are caused to perform the operations including causing the mobile device to display subindices for each of the metrics including a percentage calculation for the environmental footprint index.

[00133] Implementation 20. The system of any one of Implementations 1 to 19, wherein the one or more data processors are caused to perform the operations including causing the mobile device to display a tracker status for recycling, the tracker status including a progress bar.

[00134] Implementation 21. The system of any one of Implementations 1 to 20, wherein the mobile device includes at least two cameras for determining a depth associated with the scanned item.

[00135] Implementation 22. The system of Implementation 21, wherein the one or more data processors are caused to perform the operations including determining a depth associated with the scanned item.

[00136] Implementation 23. The system of Implementation 21 or Implementation 22, wherein the one or more data processors are caused to perform the operations including determining a volume associated with the scanned item.

[00137] Implementation 24. The system of any one of Implementations 21 to 23, wherein the one or more data processors are caused to perform the operations including determining a weight associated with the scanned item.

[00138] Implementation 25. A system for fraud detection, comprising: at least two cameras including a first camera configured to capture a first camera image and a second camera configured to capture a second camera image, the first camera image and the second camera image; one or more data processors; and a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations including: determining a depth associated with an object included in the first camera image and the second camera image using a displacement associated with the first camera image and the second camera image; and based on the determined depth being below a depth threshold, raising an alarm.

[00139] Although the disclosed examples have been illustrated and described with respect to one or more implementations, equivalent alterations and modifications will occur or be known to others skilled in the art upon the reading and understanding of this specification and the annexed drawings. In addition, while a particular feature of the invention may have been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of the other implementations as may be desired and advantageous for any given or particular application.

[00140] While various implementations of the present disclosure have been described above, it should be understood that they have been presented by way of example only, and not limitation. Numerous changes to the disclosed implementations can be made in accordance with the disclosure herein, without departing from the spirit or scope of the disclosure. Thus, the breadth and scope of the present disclosure should not be limited by any of the above described implementations. Rather, the scope of the disclosure should be defined in accordance with the following claims and their equivalents.

Claims

CLAIMS What is claimed is:

1. A system, comprising: one or more data processors; and a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations including: receiving a scanned item from a mobile device; placing the scanned item in a service queue for identification; determining whether the scanned item is recyclable; and based at least in part on the scanned item being recyclable, providing the mobile device with an indication of updated rewards.

2. The system of claim 1, wherein the one or more data processors are caused to perform the operations including: receiving the scanned item from a scene capture that includes multiple items.

3. The system of claim 2, wherein the scene capture is captured in a landscape mode of the mobile device.

4. The system of claim 2 or claim 3, wherein the one or more data processors are caused to perform the operations including: causing the mobile device to nudge a capture of the scene in landscape mode.

5. The system of claim 2 or claim 3, wherein the determining whether the scanned item is recyclable includes using an artificial intelligence model to classify the scanned item as recyclable or unrecyclable.

6. The system of claim 2 or claim 3, wherein the determining whether scanned item is recyclable includes using an artificial intelligence model to identify a material associated with the scanned item.

7. The system of claim 2 or claim 3, wherein the determining whether the scanned item is recyclable includes determining whether the scanned item is contaminated.

8. The system of any one of claims 1 to 3, wherein an amount of the updated rewards is based on a time provided recycling the scanned item.

9. The system of any one of claims 1 to 3, wherein the amount of updated rewards is based on a position on a leaderboard.

10. The system of any one of claims 1 to 3, wherein the updated rewards is based on a game.

11. The system of any one of claims 1 to 3, wherein the one or more data processors are caused to perform the operations including displaying on a screen of the mobile device information regarding the scanned item.

12. The system of claim 11, wherein the information is displayed as a quiz.

13. The system of any one of claims 1 to 3, wherein the one or more data processors are caused to perform the operations including determining whether a scanned limit threshold is reached and determining that the updated rewards is zero when the scanned limit threshold is reached.

14. The system of claim 13, wherein the scanned limit threshold is 50 items.

15. The system of any one of claims 1 to 3, wherein the one or more data processors are caused to perform the operations including determining whether a scanned limit rate is reached and determining that the updated rewards is zero when the scanned limit rate is reached.

16. The system of claim 15, wherein the scanned limit rate is 50 items in 30 seconds.

17. The system of any one of claims 1 to 3, wherein the one or more data processors are caused to perform the operations including determining an environmental footprint index based at least in part on metrics associated with a recycling practice, a use of renewable energy, a use of rideshare services, a use of electric vehicles, a use of gas or petroleum vehicles, a use of public transit, a use of eco-friendly products, a use of unrecyclable products, a support of environmentally conscious brands, donating to environmental charities, or any combination thereof.

18. The system of claim 17, wherein the metrics are combined as a weighted combination to determine the environmental footprint index.

19. The system of claim 17, wherein the one or more data processors are caused to perform the operations including causing the mobile device to display subindices for each of the metrics including a percentage calculation for the environmental footprint index.

20. The system of any one of claims 1 to 3, wherein the one or more data processors are caused to perform the operations including causing the mobile device to display a tracker status for recycling, the tracker status including a progress bar.

21. The system of any one of claims 1 to 3, wherein the mobile device includes at least two cameras for determining a depth associated with the scanned item.

22. The system of claim 21, wherein the one or more data processors are caused to perform the operations including determining a depth associated with the scanned item.

23. The system of claim 22, wherein the one or more data processors are caused to perform the operations including determining a volume associated with the scanned item.

24. The system of claim 21, wherein the one or more data processors are caused to perform the operations including determining a weight associated with the scanned item.

25. A system for fraud detection, comprising: at least two cameras including a first camera configured to capture a first camera image and a second camera configured to capture a second camera image, the first camera image and the second camera image; one or more data processors; and a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations including: determining a depth associated with an object included in the first camera image and the second camera image using a displacement associated with the first camera image and the second camera image; and based on the determined depth being below a depth threshold, raising an alarm.