The global statistics scan
This project born some time ago in a join meeting between mateslab members and a security company that was looking for a methodological way of legally obtaining statistics about services open to Internet. They needed to find out how many of certain types of services were up and running around the globe.
Goal
The goal of this project is to legally find out the amount of hosts that have a certain group of ports open.
The scanning will be service-oriented, meaning that we only look for one group of services at a time. We will not indiscriminately scan every open port.
The scanning will be service-oriented, meaning that we only look for one group of services at a time. We will not indiscriminately scan every open port.
Challenges
The challenge of the project was not to scan ports, but to scan a very large group of hosts in a timely and accurate manner. We had to design the methodology to properly generate and control the scans done and the results achieved.
Also, the scans must be legal in the countries were they are done. This means that only SYN packets were sent and only to those countries were this is not illegal.
Also, the scans must be legal in the countries were they are done. This means that only SYN packets were sent and only to those countries were this is not illegal.
Methodology proposed
We proposed to use a methodology that takes advantage of our hackspace group and speeds up the scanning process. In order to control the process and to have a good feeling about how we were going, we started with only one country.
Step 1 is straightforward using some web sites online.
- Find out the networks assigned to one country. In our case, they were more than 46,309,640 subnets.
- Find out the subnets that were worth scanning. To reduce the amount of networks to scan.
- Find out the active hosts in those networks
- Scan the specified ports in those hosts.
- Verify the performance metrics, that is, try to reduce the false positives.
Step 1 is straightforward using some web sites online.
Step 2, subnets worth scanning
To find out which subnets were worth scanning, we developed a novel technique we called NetPing. Its purpose its to find out if some network has active hosts inside or not without scanning them all. It can be considered analogous to the techniques used to find out if one host is active.
In a network with a /24 CIDR, we propose to search for some small amount of hosts. The main idea is that in any /24 network, the first hosts that are assigned are those.
To support this idea we conducted a quick 5 day test to verify this assumption. We randomly scanned 4350 networks (/24). Results are:
The top 5 hosts assigned are:
In conclusion, if you search only for the hosts 1,129,65,17,41 and 254 being active in a network, you will find out at least the 80% of the networks.
This is a huge improvement for the 80% of the networks.
In a network with a /24 CIDR, we propose to search for some small amount of hosts. The main idea is that in any /24 network, the first hosts that are assigned are those.
To support this idea we conducted a quick 5 day test to verify this assumption. We randomly scanned 4350 networks (/24). Results are:
- Total nets up: 1169 (26,87% of the total networks scanned)
- Total nets with our hosts up: 929 (79% of the total nets up, and 21.35% of the total networks scanned)
The top 5 hosts assigned are:
- Host x.x.x.1: is up in 610 networks
- Host x.x.x.129: is up in 488 networks
- Host x.x.x.65: is up in 438 networks
- Host x.x.x.17: is up in 413 networks
- Host x.x.x.41 and x.x.x.254: are up in 425 networks
In conclusion, if you search only for the hosts 1,129,65,17,41 and 254 being active in a network, you will find out at least the 80% of the networks.
This is a huge improvement for the 80% of the networks.
Step 3 and 4, find hosts up and port scan
These steps are a common nmap command. Nmap first find out if the host is up and later it scan it.
We have to use some more options in the nmap command. They were:
--open, to only store the open ports.
--reason, to help the verification process.
--max-rtt-timetout 800ms, this was because some networks with a large amount of hosts down, or hosts with a large amount of ports filtered make nmap spend more time scanning. We do not want to spend a lot of time on this, but we want to have statistically meaningful results.
We have to use some more options in the nmap command. They were:
--open, to only store the open ports.
--reason, to help the verification process.
--max-rtt-timetout 800ms, this was because some networks with a large amount of hosts down, or hosts with a large amount of ports filtered make nmap spend more time scanning. We do not want to spend a lot of time on this, but we want to have statistically meaningful results.
dnmap
It quickly become clear that one machine alone with not suffice to finish the project in time. And trying to use several nmaps at the same time would be very difficult to coordinate. That is why we created a tool called dnmap, it is a distributed nmap framework that really helped us with this.
Check out its own web page: dnmap
Check out its own web page: dnmap
Experiments
We conducted experiments during almost 2 weeks to finish our job. We had to learn a lot, fix bugs and have fun, but finally we manage to finish one country.
We still have to add more features to the dnmap, and create a automated analysis tool, but that can wait now.
We still have to add more features to the dnmap, and create a automated analysis tool, but that can wait now.
Results
Results will be published here near the 18 of March. Keep tuned!
