Table of Contents
Authors
Last year, we were sitting and discussing over coffee how the current reconnaissance techniques we used were outdated and cumbersome. We decided to do some research and improve the overall process. As a result, we set out to create a dataset of DNS Names. The service we created used multiple steps to gather all the assets for a given Domain Name. But there was a major problem with this approach - the system had lesser results than many public alternatives, it wasn’t scalable so the system couldn’t be used at scale.
The data from this service was passed to another service that ran a list of recon tools and stored the data for each one of them. Multiple type of data was stored without proper structure. There was no way to query the data in a feasible and fast manner. The quantity of data was very large and disorganized making it very hard to actually do something meaningful with it.
The Outcome of all problems
-
Check the introduction video. Today, we are really excited and happy to present our solution to the problems faced during reconnaissance process. ProjectDiscovery is a cloud based reconnaissance and continuous monitoring platform that automates the entire process of recon and provides the data in an organized and managed manner.
The aim behind building it was to solve the problems that plagued the recon process in general, improve the recon approach and make it accessible to everyone. The first and foremost issue was scalability which was solved by adopting a micro-services based architecture to execute the tasks that could scale as needed. A custom query-language was built to allow the user to manipulate the entire structured data as needed. It addresses many shortcomings of the traditional reconnaissance approach like sequential task execution, unorganized structure, fast queries on the data collected etc.
Features of the Platform
Automated Reconnaissance And Continuous Monitoring -
- Recon process is completely streamlined and simple. Configure the settings for the scan, specify the targets in scope and the system will perform all the heavy lifting for you.
- The data gathered is continuously monitored for changes and alerts are created for any change in the assets. Standalone monitors can be created for the needs of monitoring all aspects of an organization, like GitHub, Endpoints, etc.
Organized Data and Filters -
- Data is presented to the user in an organized way. Clutter is reduced and filters are provided to work with the data.
- Screenshots of hosts are provided to the users to get more visibility.
- Grouping and clustering features are provided to filter the interesting stuff and provide visibility on things that matter.
Queries and Interfaces -
- Entire data is searchable through a simple interface. All data types are indexed and made searchable.
- A query language is provided to perform complex data manipulation operations on the gathered data.
- The data is accessible via an API to allow the users to write wrapper and use the data in their own reconnaissance workflows.
- Telegram and Slack interfaces are also provided to orchestrate your reconnaissance workflows on the go.
Fully scalable and Secure -
- The platform is able to scale dynamically according to the load.
- A distributed architecture is used to allow the services to scale as needed.
- Multi-layered architecture is employed to provide an isolation layer between different parts and provide maximum security to user’s data.
Better tooling and Approaches -
- The existing tooling was highly improved upon and made faster.
- New research was done and better tools were developed to approach the problems in new ways and make the overall process more efficient saving resources and time.
Release & Goals
The service is aimed to be released in the beginning of 2020. The Private Beta invites however will start rolling hopefully by the fall of 2019.
The act of sharing is very much encouraged at ProjectDiscovery, and so the service is being kept free. Community Plan which is free is what we recommend for everyday use cases with limited number of assets, however there are some rate-limits in place to tackle abuse. Professional plan is built for users who need to monitor an excessive number of assets.
At ProjectDiscovery, we believe in the community and so, with the release of the service, we also hope to open source a lot of internal tooling and research. The Infosec community is the reason we were able to build something like this, due to their contributions and research. This is our way of giving back
Our Goal is to build a community powered platform for performing seamless reconnaissance and data collection for target and make it as easy as possible for researchers to collect and manage information about a target and continuously track the changes without doing the same tedious process manually.
All the features of the platform will be available to both professional as well as community edition users. More details about different aspects of the platform is available on the FAQ page.
Invites & Feedback -
Currently, we are starting with a closed BETA release where we’ll invite limited users to use the product and submit feedback or improvements before the public release.
You can use the Google form to request an invite.
If you have something you’d like to see in the platform, or some ideas for improvements, suggestions, etc, we’d love to hear from you. You can use the contact details below to let us know.
Contact -
If you have any questions, suggestions, ideas or concerns about ProjectDiscovery, kindly feel free to reach out to us at Twitter @pdiscoveryio or email us at contact@projectdiscovery.io. Check out our website at https://projectdiscovery.io
Let us know what you guys think in the comments section below.