NASA using Amazon SQS for NASA Image and Video Library

Amazon Simple Queue Service

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. SQS eliminates the complexity and overhead associated with managing and operating message-oriented middleware and empowers developers to focus on differentiating work. Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available. Get started with SQS in minutes using the AWS console, Command Line Interface or SDK of your choice, and three simple commands.

SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

How does SQS work?

SQS provides an API endpoint to submit messages and another endpoint to read messages from a queue. Each message can only be retrieved once, and you can have many clients submitting messages to and reading messages from a queue at the same time.

The messages that SQS handles can be unformatted strings, XML, or JSON. Because SQS guarantees “exactly once” delivery, and because you can concurrently submit messages to and read messages from a given queue, SQS is a good option for integrating multiple independent systems.

You might well be asking: why use SQS if you can have an internal HTTP API for each service? While HTTP APIs are an accessible way to expose software systems to external users, it’s not the most efficient mechanism when it comes to integrating purely internal systems. A messaging queue is more lightweight. In particular, SQS also handles things like automated retries, preserving queue states across multiple availability zones in AWS, and keeping track of expiration timeouts on all messages.

Benefits

Eliminate administrative overhead

AWS manages all ongoing operations and underlying infrastructure needed to provide a highly available and scalable message queuing service. With SQS, there is no upfront cost, no need to acquire, install, and configure messaging software, and no time-consuming build-out and maintenance of supporting infrastructure. SQS queues are dynamically created and scale automatically so you can build and grow applications quickly and efficiently.

Reliably deliver messages

Use Amazon SQS to transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be available. SQS lets you decouple application components so that they run and fail independently, increasing the overall fault tolerance of the system. Multiple copies of every message are stored redundantly across multiple availability zones so that they are available whenever needed.

Keep sensitive data secure

You can use Amazon SQS to exchange sensitive data between applications using server-side encryption (SSE) to encrypt each message body. Amazon SQS SSE integration with AWS Key Management Service (KMS) allows you to centrally manage the keys that protect SQS messages along with keys that protect your other AWS resources. AWS KMS logs every use of your encryption keys to AWS CloudTrail to help meet your regulatory and compliance needs.

Scale elastically and cost-effectively

Amazon SQS leverages the AWS cloud to dynamically scale based on demand. SQS scales elastically with your application so you don’t have to worry about capacity planning and pre-provisioning. There is no limit to the number of messages per queue, and standard queues provide nearly unlimited throughput. Costs are based on usage which provides significant cost saving versus the “always-on” model of self-managed messaging middleware.

Amazon SQS limits

Amazon SQS has a few service limits that you should consider before using SQS in production:

• Queue delay. Minimum 0 seconds, maximum 15 minutes. The built-in delay functionality in SQS queues exists to allow inserting a pause between when a message is produced and when it’s visible in the queue. If you need that delay to be longer than 15 minutes, SQS can’t offer a delay that long, so you’ll need to implement the delay in the system producing the messages before it pushes them to SQS.
• In-flight messages. SQS has a maximum of 120,000 messages (standard queue) or 20,000 messages (FIFO queue) that can be in flight: in other words, messages that have been received by a consumer but not yet removed from the queue. In most well-architected systems, it would be hard to reach this number of in-flight messages unless you are in an outage scenario. Before using SQS in production, determine if this could become an issue for you, especially in edge cases and possible error states for your application.
• Message attributes. Each message is allowed to have a maximum of 10 metadata attributes. If you’d like to have more metadata than that, consider including it in the message itself rather than as a field attached to the message.
• Message content. Messages you submit to SQS queues can be in plain text (unformatted) or in JSON or XML format. Other formats aren’t supported.
• Message retention. Minimum: 60 seconds. Maximum: 14 days. If you think you might need shorter or longer retention times, SQS might not work for you.
• Maximum message size is 256KB. If you need to send larger messages, consider uploading the larger message to S3 and including a reference to the S3 object as part of the message.
• Message visibility timeout. You might be wondering 'What is SQS visibility timeout?' The visibility timeout is a configurable time period during which SQS temporarily "hides" a message that has been received by a consumer to avoid its being processed by other consumers. After the visibility timeout expires, if the message hasn’t been removed from the queue by the consumer that received it, it becomes visible to other queue consumers. The default visibility timeout is 30 seconds, the minimum is 0 seconds and the maximum is 12 hours.

Consider all these limits for your specific use case, and take future growth into account, especially if you are building a growth-oriented system.

NASA:

"We now have an agile, scalable foundation on which to do all kinds of amazing things. Much like with the exploration of space, we’re just starting to imagine all that we can do with it.” Bryan Walls (Imagery Experts Deputy Program Manager, NASA)

About NASA

Established in 1958, the National Aeronautics and Space Administration (NASA) has been working around the world—and off of it—for almost 60 years, trying to answer some basic questions: What’s out there in space? How do we get there? What will we find? What can we learn there, or learn just by trying to get there, that will make life better here on Earth?

Exploring Space: No Rocket Science Degree Needed

Have you ever looked up at night and wondered about the mysteries of space? Or marveled at the expansiveness of our galaxy? You can easily explore all this and more at the NASA Image and Video Library, which provides easy access to more than 140,000 still images, audio recordings, and videos—documenting NASA’s more than half a century of achievements in exploring the vast unknown. For NASA, providing the public with such easy access to the wonders of space has been a journey all its own.

NASA began providing online access to photos, video, and audio in the early 2000’s, when media capture began to shift from analog and film to digital. Before long, each of NASA’s 10 field centers was making its imagery available online, including digitized versions of some older assets.

Therein was the challenge: “With media in so many different places, you needed institutional knowledge of NASA to know where to look,” says Rodney Grubbs, Imagery Experts Program Manager at NASA. “If you wanted a video of the space shuttle launch, you had to go to the Kennedy Space Center website. If you wanted pictures from the Hubble Space Telescope, you went to the Goddard Space Flight Center website. With 10 different centers and dozens of distributed image collections, it took a lot of digging around to find what you wanted.”

Early efforts to provide a one-stop shop consisted of essentially “scraping” content from the different sites, bringing it together in one place, and layering a search engine on top. “In large part, those initial efforts were unsuccessful because each center categorized its imagery in different ways,” says Grubbs. “As a result, we often had five to six copies of the same image, each described in different ways, which made searches difficult and delivered a poor user experience.”

In 2011, NASA decided that the best approach to address this issue was to start over. By late 2014, all the necessary pieces for a second attempt were in place:

• The Imagery Experts Program had developed and published a common metadata standard, which all NASA’s centers had adopted.

• The Web Enterprise Service Technologies (WESTPrime) service contract, one of five agency-wide service contracts under NASA’s Enterprise Services program, provided a delivery vehicle for building and managing the new site.

• The Federal Risk and Authorization Management Program (FedRAMP), which provides a standardized approach to security assessment, authorization, and continuous monitoring for cloud products and services.

“We wanted to build our new solution in the cloud for two reasons,” says Grubbs. “By 2014, like with many government agencies, NASA was trying to get away from buying hardware and building data centers, which are expensive to build and manage. The cloud also provided the ability to scale with ease, as needed, paying for only the capacity we use instead of having to make a large up-front investment.”

Decades of NASA Achievements – All in One Place

Development of the new NASA Image and Video Library was handled by the Web Services Office within NASA’s Enterprise Service and Integration Division. Technology selection, solution design, and implementation were managed by InfoZen (acquired by and now operating as ManTech International), the WESTPrime contract service provider. As an Advanced Consulting Partner of the AWS Partner Network (APN), ManTech International chose to build the solution on Amazon Web Services (AWS). “Amazon was the largest cloud services provider, had a strong government cloud presence, and offered the most suitable cloud in terms of elasticity,” recalls Sandeep Shilawat, Cloud Program Manager at ManTech International.

NASA formally launched its Image and Video Library in March 2017. Key features include:

• A user interface that automatically scales for PCs, tablets, and mobile phones across virtually every browser and operating system.

• A search interface that lets people easily find what they’re looking for, including the ability to choose from gallery view or list view and to narrow-down search results by media type and/or by year.

• The ability to easily download any media found on the site—or share it on Pinterest, Facebook, Twitter, or Google+.

• Access to the metadata associated with each asset, such as file size, file format, which center created the asset, and when it was created. When available, users can also view EXIF/camera data for still images such as exposure, shutter speed, and lens used.

• An application programming interface (API) for automated uploads of new content—including integration with NASA’s existing authentication mechanism.

Architecture

The NASA Image and Video Library is a cloud-native solution, with the front-end web app separated from the backend API. It runs as immutable infrastructure in a fully automated environment, with all infrastructure defined in code to support continuous integration and continuous deployment (CI/CD).

In building the solution, ManTech International took advantage of the following AWS services:

• Amazon Elastic Compute Cloud (Amazon EC2), which provides secure, resizable compute capacity in the cloud. This enables NASA to scale up under load and scale down during periods of inactivity to save money and pay for only what it uses.

• Elastic Load Balancing (ELB), which is used to distribute incoming traffic across multiple Amazon EC2 instances, as required to achieve redundancy and fault-tolerance.

• Amazon Simple Storage Service (Amazon S3), which supports object storage for incoming (uploaded) media, metadata, and published assets.

• Amazon Simple Queue Service (Amazon SQS), which is used to decouple incoming jobs from pipeline processes.

• Amazon Relational Database Service (Amazon RDS), which is used for automatic synchronization and failover.

• Amazon DynamoDB, a fast and flexible NoSQL database service, which is used to track incoming jobs, published assets, and users.

• Amazon Elastic Transcoder, which is used to transcode audio and video to various resolutions.

• Amazon CloudSearch, which is used to support searching by free text or fields.

• Amazon Simple Notification Service (Amazon SNS), which is used to trigger the processing pipeline when new content is uploaded.

• AWS CloudFormation, which enables automated creation, updating, and destruction of AWS resources. ManTech International also used the Troposphere library, which enables the creation of objects via AWS CloudFormation using Python instead of hand-coded JSON—each object representing one AWS resource such as an instance, an Elastic IP (EIP) address, or a security group.

• Amazon CloudWatch, which provides a monitoring service for AWS cloud resources and the applications running on AWS.

An Image and Video Library for the Future

Through its use of AWS, with support from ManTech International, NASA is making its vast wealth of pictures, videos, and audio files—previously in some 60 “collections” across NASA’s 10 centers—easily discoverable in one centralized location, delivering these benefits:

• Easy Access to the Wonders of Space. The Image and Video Library automatically optimizes the user experience for each user’s particular device. It is also fully compliant with Section 508 of the Rehabilitation Act, which requires federal agencies to make their technology solutions accessible to people with disabilities. Captions can be turned on or off for videos played on the site, and text-based caption files can be downloaded for any video.

• Built-in Scalability. All components of the NASA Image and Video Library are built to scale on demand, as needed to handle usage spikes. “On-demand scalability will be invaluable for events such as the solar eclipse that’s happening later this summer—both as we upload new media and as the public comes to the view that content,” says Bryan Walls, Imagery Experts Deputy Program Manager at NASA.

• Good Use of Taxpayer Dollars. By building its Image and Video Library in the cloud, NASA avoided the costs associated with deploying and maintaining server and storage hardware in-house. Instead, the agency can simply pay for the AWS resources it uses at any given time.

While NASA’s new Image and Video Library delivers a wealth of new convenience and capabilities, for people like Grubbs and Walls, it’s just the beginning. “We now have an agile, scalable foundation on which to do all kinds of amazing things,” says Walls. “Much like with the exploration of space, we’re just starting to imagine all that we can do with it.”

Thank You for Reading !!!