Dir auths getting abuse complaints about port 22 scanning

Theory one: these are relays that run on port 22, and the directory authorities are doing reachability testing to them.

But if that were so, there would be relay descriptors in the cached-descriptors* files, listing these destinations with an ORPort of 22.

There are ten such Tor relays running right now which set their ORPort to 22. You can learn them via

$ curl -s http://128.31.0.39:9231/tor/status-vote/current/consensus|grep "^r "|grep " 22 0$"

Linus has instrumented maatuska to watch all outgoing port 22 connections, and he has seen connections to the above 10 relays, but no others.

So I don't think it is theory one.

Theory two: somebody is asking our relays to extend (like, circuit extend) to destinations at port 22. This wouldn't actually let them port scan, but they could make a nuisance of themselves.

Theory two-b is that maybe there are a bunch of relays or bridges running with their ORPort on port 22 that are trying to do reachability tests yet they are not reachable, so they keep on doing the test and they never publish any relay descriptors.

We instrumented gabelmoo and maatuska and moria1 to watch where the extends are going for a while, using the attached patch, and we didn't see anything surprising.

So I don't think it's theory two or theory two-b.

(But it might still be one of these, if the scans happened during an earlier timeframe and they are no longer happening now that we're watching.)

(If it were theory two-b, we would likely hear from other non-exit relay operators that they were trying to extend to port 22 over and over too. We haven't heard this from relay operators lately; but maybe the news will start to come in soon.)

85-patch.txt

Theory three: these reported port 22 hosts don't actually appear to be listening, so there was no three-way handshake, so if the observers only saw syn packets, those could have been forged from anywhere.

That is, in this theory, some jerk is spoofing syn packets and sending them to these abuse honeypots in order to make people mad.

mentioned in issue #86 (closed)

GreyNoise has observed scanning activity matching theory 3. (requires free account to view/search)

From tor-relays - 2023-07-23:

admin at prsv.ch admin at prsv.ch Sun Jul 23 22:07:44 UTC 2023

In the past 24 hrs, I have been receiving complaints from my hosting provider that they're receiving hundreds of abuse reports related >to port scanning. I have no clue why I'm all of the sudden receiving abuse reports when this non-exit relay has been online for months >without issues. In addition, I have other non-exit relays hosted by the same provider with no issues and more across other providers.

I proceeded to reinstall the OS and reconfigure Tor. I was then quickly notified by my hosting provider again of more abuse reports >all showing port 22 as target port.

https://lists.torproject.org/pipermail/tor-relays/2023-July/021244.html

There has been a surge of reports on AbuseIPDB since October 13, 2024: https://www.abuseipdb.com/check/204.13.164.118

we got similar reports at TPA, aimed at one of the webservers in the main www.torproject.org rotation, in tpo/tpa/team#41840 (closed). What I found on the server is that we're sending lots of "reset" (RST) packets to a rather large and distributed set of IP addresses on the internet, in response to SSH connect attempts on the webserver, which doesn't allow those at all. Those are therefore backscatter RST packets, perhaps in response to spoofed IP addresses.

But my theory, right now, is that the plaintiffs are actually the source of that traffic. I wrote to cert.br and watchdogcyberdefense.com directly to try to get more information about this, before learning about this ticket.

so, to reframe this a set of theory completing @arma's list:

theory four: we are actively, actually sending TCP RST packets to those destinations, which are either (a) compromised hosts actually trying to scan us or (b) spoofed IP addresses and the victims are just getting the backscatter

Unless proof of other behavior, i would tend to think 4a is the valid theory right now... This something that can be checked on an affected host with:

tcpdump -n port 22 and '!' host YOUR_IP_HERE

You should be seeing traffic like:

18:22:01.899153 IP 149.87.158.215.22 > 116.202.120.165.23990: Flags [S.], seq 82678680, ack 1276858598, win 64240, options [mss 1460], length 0
18:22:01.899173 IP 116.202.120.165.23990 > 149.87.158.215.22: Flags [R], seq 1276858598, win 0, length 0

In our case, the firewall is the one bouncing the requests, but depending on your configuration, something else might be refusing the connection.

I'm building a trace of all the hosts we're sending RST packets to, happy to share:

tcpdump -n dst port 22 and '!' dst host 116.202.120.165

so far i'm in the thousands of hosts.

I spoke to anarcat offline, and he was indeed reading his tcpdump wrong, and we are back to theory three like we have thought. (More details about that confusion on the tpa team ticket about the webserver side.)

added Doing label

assigned to @arma

mentioned in incident tpo/tpa/team#41840 (closed)

marked this issue as related to #86 (closed)

Cross-linking https://lists.torproject.org/pipermail/tor-relays/2024-October/021953.html where I had the same discovery on the 4 relays I run (geo- and ISP-diverse) and went independently through the investigation process. Strongly corroborates theory 3: seeing both SYN-ACKs and RSTs randomly coming at a low rate (few per minute).

mentioned in issue tpo/community/relays#103 (closed)

I'm running a non-exit relay at Hetzner, and yesterday got an abuse report (from CyberWatchDogs) as well, for what appear to be outgoing SSH connections. Note that it's the same /16 block as your reports.

abuse_report.txt

It was late at night and I panicked, so I nuked the server from orbit. It was performing other duties (Rustdesk relay, ...) as well, and I thought it had been compromised, so I have no additional forensic data.

I have not received a complaint from my ISP (Free SAS) yet, but I am also recording unsolicited SYN+ACK and RST from port 22 of IPs i have no business interacting with.

I am also recording unsolicited SYN+ACK and RST from port 22 of IPs i have no business interacting with.

I'd be curious to hear how you're recording those, feel free to reach out of band if uncomfortable to do it publicly.

i'm using this tcpdump rule (the ip being mine) ((tcp src port 22 and not src 82.64.238.84) or (tcp dst port 22 and not dst 82.64.238.84)) and tcp[tcpflags] & (tcp-syn|tcp-rst) != 0', and I look with my eyes for S. and R. with no previous S.

i would tentatively suggest to people to not drop the incoming SYN+ACK packets as well. feel free to not log them if that is a problem, for example, but returning a RST packet to the victims might actually help them in dealing with the attack, because then they don't need to keep track of all those open connections.

happy to get confirmation or NACK on this idea: we're not dropping outgoing traffic right now.

Got complaints regarding this from 3 different providers.

But all 3 had the same 202.91.162.0/24 as DST (Port 22) and in the complaint noc@watchdogcyberdefense.com was given as a contact option.

Maybe this was always a problem and just now some AI "enhanced" software is sending out automated abuse reports?

Another one with same /24 https://lists.torproject.org/pipermail/tor-relays/2024-November/021988.html

Personally got abuse mails from nearly all of my providers now. Sometimes multiple times from the same. An official statement would be really nice, so the providers are more willingly to just ignore everything coming from noc@watchdogcyberdefense.com and 202.91.162.0/24, where all the reports for me are originated. One of those providers shut down 4 servers and closed my account. I'm running non exits so I normally do not have to handle abuse…

Ok, wrote to noc@watchdogcyberdefense.com and even got a quick reply. They are considering the IPs of the tor nodes compromised, because they mostly show up on virustotal as "malicious".

Tried to explain, this does not mean they are infected by malware, but not sure if this will help…

Ok, wrote to noc@watchdogcyberdefense.com and even got a quick reply. They are considering the IPs of the tor nodes compromised, because they mostly show up on virustotal as "malicious".

Tried to explain, this does not mean they are infected by malware, but not sure if this will help…

They made a similar weird argument about the www.torproject.org web server, I have no idea what that's about.

It would make sense if exit relays got marked as malicious but for non-exit relays to get shutdown by providers over baseless claims is ridiculous.

Got a new complaint today. This is what I replied to Hetzner

Hey,

I think someone is spoofing Hetzner's IP range. The Tor community has some analysis:

https://gitlab.torproject.org/tpo/network-health/analysis/-/issues/85

https://gitlab.torproject.org/tpo/community/relays/-/issues/103

Extensive analysis in this blog: https://delroth.net/posts/spoofed-mass-scan-abuse/#content-start

There isn't a lot I can do. Someone is abusing your IP range to knock on SSH ports on the internet, possibly to get your IP's flagged or blacklisted.

Please look into this more closely - I'm happy to assist where I can.

Jeroen

Based on what is being reported on mailing list. It looks like operators are starting to receive abuse complaints from non-Hetzner providers. I have just received one from one of my providers today from the same watchdog group and my provider are threatening to null route my IPs

Also I noticed that Gus attached an email I made in 2023 regarding reports of abuse complaints. I would like to note, I ended up cancelling my server with that hosting provider due to these issues.

Since that incident, I have also received reports from one of my other hosting provider of DNS AMP attacks coming from one of my non-exit relays too. According to my provider, they received reports for about 2 days and then it stopped.

I hope there is some resolution to this ongoing issue.

Edit: I just checked the IPs that were reported to be port scanning on abuseipdb. There is no reports found on abuseipdb or any other database. Are we getting fake abuse complaints??

Update regarding the abuse complaint I received.

My hosting provider has forwarded my response calling it a false report and that someone is spoofing IPs.

There is currently no response back from this watchdog group and my hosting provider has not taken any action against me.

I have received multiple notices from OVH EU. So far only 2 of my providers have forwarded notices. I have nothing from the others including OVH US. I'm guessing every provider got this notice. It seems some providers may have just ignored it or delayed notice.

Edit: I also found it interesting in the OVH forwarded notice. The email for the watchdog has been removed.

Your prompt attention to this matter would be greatly appreciated. We value your expertise and cooperation in resolving this situation effectively. Thank you for your time and consideration. 
For any corrections/updates, kindly email email-removed@provider[.]com& lt;/pre></body></ html> 

-- end of the technical details --

marked this issue as related to tpo/tpa/team#41840 (closed)

Did anyone with a proper torproject.org email-address already contact noc@watchdogcyberdefense.com or maybe even better the CEO "Roger Do" <roger (at) qsearch (dot) cc> directly? "AI Managed Security Service" needs some human intervention I guess…

(IONOS already shut down 13 nodes [369.58 MiB/s] https://metrics.torproject.org/rs.html#search/host_name:ketamin.trade / https://lists.torproject.org/pipermail/tor-relays/2024-October/021976.html and personally I know about a similar case with a different provider)

Did anyone with a proper torproject.org email-address already contact noc@watchdogcyberdefense.com

I did, in response to a Hetzner complaint. They claim they are implementing BCP38 and properly rule out spoofed packets, so my current theory is that spoofed traffic might be coming from inside hetzner.

If someone has contacts at Hetzner, it would be the best place to ask, IMHO.

We need someone on edge routers to look at that traffic and figure out where those SYN packets are generated.

That said, we (TPA, torproject.org machines), seem mostly unaffected by this: we received only one somewhat threatening letter from Hetzner (basically asking us for a "reply or else"), we replied, and haven't been bothered by them since.

They claim they are implementing BCP38 and properly rule out spoofed packet

Even if they do comply to BCP38, that doesn't mean they can rule out spoofed packets, since they could be coming from their transit which would obviously be allowed to carry traffic from any source.

Even if they did comply to BCP38, that doesn't mean they can rule out spoofed packets, since they could be coming from their transit which would obviously be allowed to carry traffic from any source.

Of course.

I can't reconcile this "of course" with "my current theory is that spoofed traffic might be coming from inside hetzner"...

Especially since at this point we've also seen that company sending abuse reports to plenty of other ISPs than just hetzner, e.g. the recent Verizon FiOS report forwarded to nanog@.

i'm sorry, it seems we have a misunderstanding.

I mean to say that they claim to implement BCP38 properly. The actual quote is:

We do practice BCP38 AND have DROP rules on sensitive ports like SSH(22) AT the edge. These packets are silently dropped (not rejected). It never gets to destination IPs. So that being the case, it is unlikely that our devices are responding to spoofed packets.

I agree with you that it's quite possible something else in their AS path might be spoofing packets.

It would be interesting to see which AS the spoofed traffic is coming from, and what they think is going on as well. I've actually suggested they investigate in that way, but so far haven't received a response.

the recent Verizon FiOS report forwarded to nanog@.

oh this is interesting though, I have been wondering about reaching out to nanog about this... is this the message you are refering to?

https://mailman.nanog.org/pipermail/nanog/2024-October/226471.html

it would be really nice to have help from some BGP-aware people to trace the AS generating that garbage and make it stop... i'm not super active in nanog anymore, so i'm perhaps not the best for that...

Dear operators, feel free to use the following template to respond to your provider. Simply copy it and replace the relevant information in the brackets.

Hello,

Thank you for contacting me about the issue described in [AbuseID] concerning my server in your datacenter with IPv4 address [IP].
I apologize for the inconvenience these reports are causing.

This particular server operates as a Tor relay: https://metrics.torproject.org/rs.html#details/[FINGERPRINT], and I've confirmed that the host is secure and configured as a non-exit relay, meaning it doesn’t originate external traffic.

I have monitoring systems in place to detect any potential issues, such as a compromise.
In this case, we believe the abuse reports are due to an attacker spoofing SYN packets with Tor relay IPs as the source, targeting random destinations.
This method generates automated abuse complaints to organizations like yours, likely as an attempt to disrupt the Tor network.

The Tor Project is actively tracking this issue in this public ticket:
https://gitlab.torproject.org/tpo/network-health/analysis/-/issues/85

Additionally, other Tor relay operators have documented similar incidents online: https://delroth.net/posts/spoofed-mass-scan-abuse/

Thank you for your support and understanding.

Best regards,
[Your Name]

changed the description

Hello everyone,

I'm writing to share that the origin of the spoofed packets has been identified and successfully shut down today, thanks to the assistance from Andrew Morris at GreyNoise and anonymous contributors.

I want to give special thanks to the members of our community who have dedicated their time and efforts to track down the perpetrators of this attack.

Although this fake abuse incident had minimal impact on the network -- temporarily taking only a few relays offline -- it has been a frustrating issue for many relay operators. However, I want to reassure everyone that this disruption had no effect on Tor users whatsoever.

We're incredibly fortunate to have such a skilled and committed group of relay operators standing with Tor.

Thank you all for your resilience, ongoing support and for making the Tor network possible by running relays.

closed

Great! Can confirm the flood of SYN ACK and RST packets arriving at my relay dropped significantly. Thanks for all the efforts.

Can you share more details about the origin of the spoofed packets?

Are there any plans for adding a filtering to avoid this in the future?

Adding a filtering to what?

The problem is that the targets of the attack aren't generating the traffic (somebody else is generating it, elsewhere on the internet, pretending to be the targets), so it doesn't help for the targets to try to filter anything.

This is a bigger issue than just for Tor. The problem is that the internet protocols allow computers to pretend to be other computers on the other side of the planet. The fix is that every ISP needs to implement what is called egress filtering, to stop themselves from sending traffic that isn't from their IP address space. So in that sense yes, filters do help, but everybody all around the internet needs to deploy them.

For now, the fix if somebody tries the attack again is to use our friends and partners who help run the internet to track down where the spoofed traffic is coming from, and shut it down again.

It's not just up to ISP to implement egress filtering. Individual applications can easily add it as well.

I feel like there is a missunderstanding about what is happening, and how to remediate the problem. Here is a quick doodle of what is happening (except there would be a lot more intermediate in the real world), where filtering is useful, and where it isn't, and why:

egress filtering is something tor does, it's called ExitPolicy, however it is useless at preventing this kind of situation because it isn't on the path of the packets that should be dropped. It's the leftmost blue mark on my doodle

Update: https://blog.torproject.org/defending-tor-mitigating-IP-spoofing/

Today I found my VPS at HostSlick shut down. After restarting it showed "network suspended", so I opened a ticket, asking what happened. (I never received an abuse report before) They answered:

Reason is malicious Traffic was coming from this vm and causing Packet loss to whole node and therefore damages to us. It Looks for me like some infected tor node or Something. There was also jetdirect packets from this Which is basically print Traffic

I replied with the link to the blog post, then they asked me:

Can you fix this then?

I asked them, what they want me to do. They replied:

Anything so we dont have Trouble Here with our customers on Same Server complaining.

I offered them to just set up the relay from scratch or providing any network dumps. No reply to that yet, can't "fix this" if I don't have access to the VPS.

Given that I didn't receive any details, I am really confused about the jetdirect packets? I also wonder if they just ignored former abuse reports til now or if there is some new stuff going on.

Dir auths getting abuse complaints about port 22 scanning

Designs

Child items 0

Activity