Government and Regulation Archives

I have much criticism about the Israeli government information security status and activity. Lots of PR and marketing for being a “Cyber Nation” but it is true mostly for the selling of knowledge, services and products. Not so much doing the “dirty work” of protecting the Israeli Internet facing IT, including the one of the Israel government itself.

So, I started this weekend to operate a mini-project of checking the quality of HTTPS sites of the gov.il sub-domains. I publish the results in a public spreadsheet, accessible as read-only (viewing, downloading and printing is allowed anonymously) from Google – https://docs.google.com/spreadsheets/d/1Fez2A1FwqesWRuc2DowpdG20okh-KDbYfO5qnl8nZ-U/edit?usp=sharing

While running these checks I found something alarming:

One of gov.il’s DNS servers, a server called dns3.gov.il, at 62.219.20.20 – replied at port 22 TCP, i.e. SSH, a service used to manage mostly Unix/Linux servers. Yes, open, allowing me try to login…

Telnet access to port 22 TCP to the server

Access to the server using PuTTY, the digital certificate prompt

As I found it, in the afternoon of Friday, 31-Jan-20, I sent this finding at 16:28 to the Israeli national CERT and after ten minutes I also managed to correspond with someone I am in contact with, who was senior in government’s information security department, and he forwarded this issue to another security senior in the government’s IT, and he passed the details on to those who can fix this issue.

It took a few hours, but at the last check I did, around 21:30 that evening – access was blocked as this port has been closed, so it seemed to me that the issue had been addressed. Quite fast for almost Friday evening (the beginning of Shabbat in Israel).

Another point was that the SSH server identification was SSH-2.0-OpenSSH_7.4p1 Debian-10 + deb9u6, which means that the OpenSSH version is 7.4p1 that was released in December 2016, and since then several versions and a few security fixes for various security issues were released for OpenSSH, which are probably missing now from server. I hope they will update what is needed as soon as possible.

OpenSSH release notes
https://www.openssh.com/releasenotes.html

OpenSSH vulnerabilities
https://www.cvedetails.com/version-list/97/585/1/Openbsd-Openssh.html

The network capture of the PuTTY access to the server

I am not a Zendesk expert but I have seen enough. Here is my story.

The short version:

If in your ZD settings the check box of “Require authentication to download” (in the site path of Admin > Settings > Tickets) is NOT selected (hence Disabled) – there are web address/URLs at sub-domain sites of zdusercontent.com that store files that are accessible without any authentication, anonymously, and they can be files that hold private data of your customers.

This check box is disabled by default! By a ZD intentional decision. Customers of ZD may not even know that and not change this default and thus the files will remain accessible anonymously. Not nice. I guess not GDPR compliant.

ZD support article about this check box – https://support.zendesk.com/hc/en-us/articles/203927716-Attachments-in-Zendesk-Support#sec5

Some of this data is indexed by Google, sample searches:

site:zdusercontent.com

https://www.google.co.il/search?q=site%3Azdusercontent.com

site:zdusercontent.com receipt

https://www.google.co.il/search?&q=site%3Azdusercontent.com+receipt

And so on – try the words like bank , “credit card”

Also, if you go to the following link you can find who ZD customers are

https://www.zendesk.com/why-zendesk/customers/

And then you can search by their names, Say, Uber

site:zdusercontent.com uber

https://www.google.co.il/search?q=site:zdusercontent.com+uber

These URLs are quite long and use complex and random characters, so they are not easy at all to guess. But, they can be sent to your customers from the ZD system as links in emails (which can be exposed in many ways) or they can be logged in your security systems, hence exposed to your IT team (see the longer version of this story below).

Since these URLs are accessed anonymously, I guess the only possible way to track who used them is by source IP, which of course can easily not be the real IP of the person who initiated this access (say if the person is using a proxy server or public VPN service).

So, my recommendation to you is to enable this check box, which will change this behavior and any access to any attachment file will force the accessing person to first be authenticated by the ZD system.

This may have negative operational results for the ease of your customers’ access to these files – so weight the pros and cons before doing this change.

The long story:

One day last week, as I was reviewing our gateway alerts, I noticed a strange link, beginning with a sub-domain of zdusercontent.com and followed by medium size string of a URL parameter. I searched to find who is the owner of this domain and found it is owned by ZD. Cool. Safe. I clicked it.

A JPG file loaded into my browser. It was a photo a customer of ours took, to prove his identity, a personal identification document… whoaaa… what???

Although I knew I didn’t log into ZD recently, I cleared my browser’s cache and all cookies, and tried again. The same…

I tried using another browser. The same.

I tried from another PC inside the company. The same.

I tried from my private mobile phone, I tried from my home. All the same.

I tried another link found for this domain – a zip file with multiple files sent by another customer. Not nice, not at all.

Woooo, I said to myself, we’re going to make tons of money on this one via a bug bounty. Zero authentication for customers’ private data. No joke.

So, I found ZD bug bounty page at HackerOne – https://hackerone.com/zendesk

It didn’t mention that the domain of zdusercontent.com is included in the bounty program.

I didn’t give up – I asked HackerOne about it, but quickly I learned HO is not really responsive nor knowledgeable so I turned directly to ZD security.

They promised me that zdusercontent.com is included in the bounty and that they wish to accept my report. (BTW, even now this domain is not mentioned as eligible to their above bug bounty page).

So I PGPed my findings and sent it to ZD, including an offer to simply block search engines from indexing these sites with a simple robots.txt file.

They replied:

”

To summarize the issue you reported to us, you found files (Zendesk ticket attachments) were indexed by the Google search engine and could be accessed publicly, without authentication, and in some instances without the token parameter in the URL. If I have missed anything please correct me.

This specific topic is something which has been brought to our attention before and has been discussed internally. I want to assure you everything is currently working as intended. All Zendesk accounts have an admin setting to require authentication to view/download a ticket attachment: https://support.zendesk.com/hc/en-us/articles/203927716-Attachments-in-Zendesk-Support#sec5. If you are worried about potential information disclosure please enable that setting to restrict access to all ticket attachments, including the files which are indexed by search engines. In that page you can see the only time ticket attachments are indexed by search engines is when the links are posted on third-party public websites. The files being indexed are not being leaked from Zendesk, but intentionally posted to public locations. This setting exists because the feature of publicly shareable ticket attachments are a popular request from our customers. That being said, I completely agree with you that there is no reason to not include a robots.txt file for that domain. There is currently an open request to implement robots.txt on a few domains which handle customer attachments which should be rolled out by the end of the year, if not much sooner.

In the meantime, if enabling the “require authentication” for attachments doesn’t fit your organization’s needs, please take a look at our Attachments API which would allow you to handle attachments on an individual basis. You can redact comment attachments via the information provided here: https://developer.zendesk.com/rest_api/docs/core/attachments#redact-comment-attachment. You can permanently delete uploads via there information provided here: https://developer.zendesk.com/rest_api/docs/core/attachments#delete-upload.

”

And their next response after I replied to the above with amazement to their answer and asking if this check box is disabled by default:

”

The administrative setting of “Require authentication to download” is disabled by default. Many of our customers specifically ask for the ability to host and share non-sensitive documents with their customers so we give them the ability to configure their account to best fit their needs. Additionally, many of our customers’ customers utilize Zendesk strictly through e-mail and not the actual Zendesk UI, therefore they wouldn’t have a registered account to begin with which could cause a lot of confusion if the e-mail correspondence contains attachments. That being said, I’ve escalated this to my manager to start the discussion with our Product teams regarding the default nature of that setting. I can’t promise a change as this may be an accepted risk using the shared responsibility model.

I’ve already mentioned the API endpoints available which Zendesk accounts can use to redact/delete all non-inline attachments. If you are worried about any specific attachments I would reach out to that specific account and address the issue with them. If they are unsure how to proceed with that type of request Zendesk is more than happy to walk them through that process.

”

I think ZD is making a mistake by disabling this check box by default, loading on itself the legal responsibility for data exposure, when some of it may be private.

If they enabled it by default – they would have been covered themselves better, making the default more secure and if customers would try to change it – they would be displayed with a flashing bold warning about the consequences of such change, and if they do change it – they will be the ones responsible for any relevant data exposure.

So, there goes my imaginary pile of bug bounty money, but as least I came across a good story and a chance to let you know about this risk and possibly mitigate it.

A next day addition I forgot to add – I found only one place on the web that already related to this issue – and it is from a poker forum, where a customer complaints about his personal data being freely exposed on the web, in a zdusercontent.com sub-domain. And he is furious… – https://forumserver.twoplustwo.com/252/global-poker/security-issue-personal-documents-posted-open-web-1715425/

(4-May-2020 – I added here the above forum discussion as a PDF file, for archiving – zd-angry-customer)

Addition at 22-Nov-18: Hi Zendesk folks, I see you reacted quickly and cleared the search results from Google. That’s very good. Just remember there are more search engines you need to handle, some main ones:

Bing – https://www.bing.com/search?q=site%3Azdusercontent.com