Proxies, ethics, and FBI
Welcome to our monthly interview, this time it’s the turn of Neil Emeigh, CEO at Rayobyte, a leading company in the proxy industry.
Hi Neil, thanks for spending some time together to share with us your thoughts. First of all, tell us more about you and what brought you to be the CEO of Rayobyte.
Hi, Pierluigi! It’s a pleasure to speak to you today.
I started using proxies when I was 14 years old, mainly for SEO purposes. As I used them I kept running into problems with sketchy providers who had poor uptime and bad customer service. Providers who I knew weren’t running their business in an ethical manner – or even one that was safe for their customers! I knew I could do better than that, so I did 🙂
Our company was originally called “Blazing SEO” – Blazing because our speed was our first differentiator, and SEO because, like I said, SEO was my main focus at the time and the main thing I knew I could sell proxies for. That name stuck a lot longer than intended – these days, ban rates are much more important than speed, and as one of the largest proxy providers, our company does a lot more than SEO.
So in July of this year, we officially rebranded as “Rayobyte” – a name that basically means “the largest finite amount of bytes possible”. This name better reflects our promise to customers today, which is that no matter how much data you’re looking for, how much bandwidth you require, or how many IPs you need.
What’s your feeling about the direction of the proxy industry? Are we in a consolidation phase or there are still new players popping out every day?
I certainly hope we’re not in a consolidation phase – that would be so boring! Fortunately, I don’t think we’re quite there yet. Proxyway says that in 2021 proxy traffic increased by 60-70%, so there’s still plenty of room for new providers. And you still see new companies like Soax and IPRoyal, which entered the market in 2019 and 2021 respectively, making a name for themselves.
One thing I want to say here is that we as proxy providers have a choice in what the market looks like. I’m not here to name names, but there are some providers out there that are making it harder for companies to enter the proxy market with aggressive legal action, and others that drag down the reputation of the industry with their shady tactics.
Liked the article? Subscribe for free to The Web Scraping Club to receive twice a week a new one in your inbox.
I really wanted to do the opposite with my recent Extract Summit talk, which you covered on your site – I wanted to share what it actually takes to start a proxy business in the modern market. Hopefully, some people see that and use the information to start new businesses of their own. I also have to mention that Rayobyte has a reseller program and welcomes anyone to use our proxies to start their business – of course, this is a money-making venture for us, but I also believe that an ethical, open data scraping movement can’t start with just one company.
From the outside, it seems to me a really hard business, where it’s difficult to stand out. From a business perspective, what’s the key feature of a successful proxy company? Is it all a matter of prices or for being accredited as a trusted partner, there should be some more values to share with your customers?
Well, it’s a lot of things. Obviously, I was lucky in that sense to get in on the “ground floor” when the more legitimate companies – the big names we know today, including Rayobyte – were just starting out. If you’re joining today you have a lot more competition to deal with, and competition that actually looks like and operates like a real business – not just some sketchy guy on a forum delivering you proxies through a Gmail account.
That said, I do think it’s easier than it might appear from the outside. For one thing, most of the large companies out there that are using proxies for data scraping don’t want to trust just a single provider, because the risk of bans is so high. To be honest, most of our largest customers are open about the fact that they’re also buying from our competitors, so as not to put all their eggs in one basket. So there’s always a market for more reliable providers out there because one company can’t dominate an entire segment or even an entire customer on its own.
Moreover, a lot of providers are specialized in one type of proxy. Rayobyte, for example, has the best data center proxies in the biz – don’t take my word for it, look at the success rates for yourself. And in fact, until last year, the data center was all we had (because we weren’t sure we could find an ethical way to source residential IPs). Data center proxies get comparable results to residential proxies in a LOT of use cases – not all, but enough for us to become a multimillion-dollar company. And they’re both cheaper and faster, which gave us an instant competitive edge for those use cases. These days data center proxies are going out of style as more websites learn how to ban them, but you could be the best mobile proxy provider, the best residential proxy provider, or the best rotating ISP proxy provider – hone in on one thing and become the best at it as we did!
As for what is specifically the key feature of a successful proxy company, it’s this: how reliably can you provide results for your customers? Yeah, proxies are seen as a kind of a commodity, which means the price is a big factor – but if you’re only looking for the lowest possible price, you’re going to end up getting what you pay for. We have a business because we have great service, which means not only that our reps are kind and well-informed, but that we actually go above and beyond to set our customers up with what we know are the best mix of proxies for their use case because ultimately we’re the experts.
What you gotta understand is that nobody wants to buy proxies – they want to buy scraping success. So you have to provide them with the IPs that are going to give them the lowest ban rates, the fastest results – give them those successful scraping projects as quickly and reliably as possible. If you can do that, you’ve got a customer for life – even if you’re charging a little more than the average. And that’s not some kind of proprietary rocket science – it’s just the sourcing stuff I talked about in my Extract presentation.
What I really appreciated both from your speech at Extract Summit and from your website is your emphasis on ethics. Can you explain what it really means in your business using ethics related to proxies?
There are two concerns when it comes to ethical proxies: the first is usage, and the second is an acquisition
Ethical usage means: what are people using your proxies for? When most people think of proxies, they think about hacking. They think about someone sneaking into your network to steal private data. They think about botnets.
What they don’t think about is that Google makes all their business from scraping web pages. Pretty much every eCommerce company is using scraping research. That sites like Letterboxd or Goodreads only exist by scraping databases of movie and book data. So much business is built on scraping because, at the end of the day, scraping is just research! It’s just looking at competitors’ websites or Facebook pages or Google listings, except instead of looking at them one at a time you get a bot to do it at scale and just share the relevant info with you.
You know this, of course, and I’m sure your readers do too – but that’s the idea we have to change. And it starts by just, you know, not letting your users do sketchy stuff. Putting in the automated AND manual checking systems will ensure nobody can access private data using your proxies. As you say, people can read all about this on our website.
As for ethical acquisition, this mostly applies to residential proxies. In case anyone doesn’t know, these are the IP addresses that come from actual devices – your ordinary home computer, cell phone, or fire stick. These are in high demand because they’re almost impossible to ban en masse, but our industry has often obtained these IPs by tricking their users into signing a term of service that basically says “we can use your IP address for whatever you want.” Or worse, not even doing that – just using a script to take IPs!
Obviously, we don’t do that. Everyone in our residential proxy pool knows that their IP address is being used as a proxy and gets reminded of that every month. They can opt-out at any time and we only use their IPs if their devices are connected to WiFi, plugged in, and not in use. Plus, we pay them for it through our Cash Raven app. So it’s a much more ethical approach – no one’s being fooled, and no one’s giving anything away for free.
Some vendors give free SDK to app developers so that when they publish their app, they can monetize from the proxy traffic generated by every app installation on the final user device. Apple recently updated IOS to alert users about third-party tracking and it has been a shock for mobile ads business. Do you think that is feasible or quite near in the future a similar move also for “third-party connections” inside apps? How big would be this change?
As a matter of fact, we have the same type of SDK, though it’s Android-exclusive. For that reason, I have to admit that I’m not an expert on the iOS store or its current policies. Frankly, they’re a pain to develop for.
That said, my non-expert opinion is that yeah, I think some kind of crackdown on proxy SDKs is totally possible. Our industry doesn’t exist in a vacuum, and neither Google nor Apple is very fond of other companies using proxies or data scraping. If the proxy industry can’t get the stink of unethical behavior off of us, we’re going to see a lot more moves to regulate our current modes of operation from both the public and private sectors. Whether such a move is going to happen any time soon, I couldn’t say – I haven’t heard anything that suggests it’s on the horizon, but it wouldn’t shock me.
I’ll say this: if it was a change that simply made people more aware that their devices are being used as proxies, I’d be all for it. As I said, we already remind our residential pool every month that their device is being used in that way, so it wouldn’t hurt us. If your business model can only exist through tricking people, burying their “consent” in a giant TOS they’ll never read, well, I won’t lose sleep over that sort of business model going away for good.
Technically speaking, instead, what’s the hardest part for your company? Keeping the IPs clean from the blacklists? Avoiding users misuse your services for illegal purposes?
It’s diversity, no doubt. This is something I talked about a lot in the Extract presentation -bans are going to happen, either way, so you need IPs from a ton of different sources to make sure they don’t all get banned at once, and your customers’ scraping projects can proceed without much interruption.
Finding a lot of IP sources isn’t too hard, but finding quality IP sources is a lot harder. Especially with our ethical standards. Let’s just be honest: there’s a reason some companies have turned to shadier acquisition methods – it’s just easier than finding people who are willing to let you use your IP as a proxy. That’s a trade I’ll make every time, but it certainly makes finding a lot of IP sources harder.
Then once you find high-quality sources, you have to ensure that your clients don’t misuse them to the point of damaging that quality. By “misuse” I don’t mean doing illegal or unethical stuff, which we’d obviously never allow in the first place, but rather just hammering the same subnets and ASNs over and over again until they’ve been banned from all the big sites.
Fortunately, we have a world-class inventory team, and they do a great job working with our customers to get them the best IPs and make sure they spread their usage out over a lot of sources.
I am pretty sure you have some great stories to tell us about your career in this industry. Would you mind sharing some of the funniest, strange, or most insane?
Here’s my craziest story. The year was 2018. Our tech director, Taylor Fleeman, and I had just recently moved from our tiny Nebraska hometown to a remote work environment in North Carolina. At the very considerate hour of 7 AM, we hear a knock at the door. To my surprise, the people knocking turned out to be two FBI agents with their badges out, guns locked and loaded.
They asked if they could come in. They didn’t have a warrant, so I should have said no – I found out later that the whole reason they were coming to us at 7 AM is that they knew that our lawyers probably weren’t getting into the office until 9, giving them plenty of time to intimidate us. But I was young, and the limits of jurisprudence weren’t exactly the first thing on my mind when I saw two men with guns show up at my home, so I just said “Absolutely, sir, right this way.”
Now I’m sure you’ve heard of the term “good cop, bad cop” from movies and television…well, I guess these guys had seen the same movies as you because they were playing that up to an almost comical degree. One guy was very aggressive, threatening a ton of prison time, while the other kept telling him to calm down and kept saying stuff like “Things will be okay kid, just help us out and we’ll be out of your hair.”
They asked us a lot of questions about certain IP addresses, what they were being used for, the credit card numbers of our company, and so on, and after I answered all our questions they left. Later I found out that the New York branch of the FBI had a case involving a hacker using one of our IPs and they wanted to know where I lived, but they couldn’t find me since we’d recently moved. So they called all around my hometown until they finally heard someone say “oh, Neil is in Charlotte!” And that’s how the Charlotte FBI came to my home.
Any advice for someone willing to create today a brand new proxy company?
Practical advice: Start by leasing or reselling IPs rather than buying your own, and focus on acquiring diversity and flexibility. This includes subnet and ASN diversity – you’ll differentiate yourself right away if you simply have IPs on two ASNs instead of one. And avoid AFRINIC IP addresses like the plague – sure they’re cheap, but they don’t work for your customers.
Philosophical advice: Be prepared to work with your clients to find the right mixture of control and diversity for their use cases. Being a great proxy provider means understanding what your customers actually want to do with those proxies, and what they need to be successful. I’m not sure I would be a successful proxy provider today if I hadn’t started as a proxy user, with an intimate knowledge of what the customer needed because, after all, I was the customer!
Is any of our you working on something spectacular in web scraping and want to share with us? please write to firstname.lastname@example.org and you could be in the next interview!