The I.T. Girl

by Pearse, Fiona

  THIS NEEDS TO BE FIXED NOW!!!

  HOW HARD IS IT TO GET A DATA FEED TO WORK PROPERLY????

  I tried to answer the complaints and update the bug report, offering a dash of possibilities while taking a closer look at the log file, checking for error messages or any pattern correlating with the missing data. There was none.

  ‘Talk to me!’ Boris was back.

  ‘Networks are on the case. In the meantime, look at this,’ I showed him the figures.

  Memory 40%

  CPU 10%

  ‘These are statistics on the live server.’ I explained. ‘That memory usage looks too high. You wouldn’t expect more than about fifteen per cent.’

  ‘Does that mean there could be a memory leak or something?’

  ‘Possibly.’ I could feel panic rising.

  ‘Can you recreate this in your test environment?’

  ‘I’ll try it now.’

  ‘Keep me informed.’

  I started my feed in my test environment and fed it sample market data. Then I ran performance analysis, hoping it would detect a memory leak or any sign of corrupted memory. But the results came back normal.

  ‘Can I help?’ Sam came round behind me.

  ‘I’m running performance analysis but nothing’s come up as a red flag. Without a large amount of data to run the test with I can’t reproduce the same memory behaviour.’

  ‘What are you seeing on the live server?’

  ‘Memory is high for some reason.’ I stared at the figures, leaning into the screen. I could feel the heat from the monitor on my face. ‘But the number of packets being processed is reasonable. The server shouldn’t be under any strain. I don’t understand.’

  ‘Some kind of build-up then.’

  He had a point. ‘Maybe there’s a non-virtual destructor somewhere.’

  Another email popped up on my screen.

  Felix Stern to Orla Hanlon

  See me now

  I jumped back. ‘Is he going to give me a hard time?’

  ‘He’s not going to give you flowers and a trophy. He’s just got to make sure everything’s being done to fix the problem.’

  ‘Bloody hell,’ I said standing up.

  ‘Give me a shout later if you need me.’

  ‘Thanks, Sam.’ I tapped his arm as we both left my cube.

  I kept my eyes down and fiddled with my hair as I walked around to the meeting rooms. By now, everyone knew METX was having problems and everyone could guess I was on my way to talk to Felix Stern. I had never actually met Felix. My only impression of him was from the bowling competition. I had just joined the company and entered for the fun. I came out of the match in an early round after one embarrassing throw where the ball went right off the green. I then spent the rest of the afternoon wandering around the other matches, trying to remember people’s names. I’d got an ice cream for the final and accidentally stood next to Sam. I remember we hadn’t spoken yet. He didn’t usually join in the social events and he was always so serious in work, as if he wanted the distance.

  ‘Sam, how’d you do?’ I finally broke the ice.

  ‘Out in the second round,’ he said evenly, with his eyes on the match. His face was flushed from the unexpected heat and his hair, usually neat, was damp with dark bits sticking to the side of his face. ‘Didn’t do as badly as you though.’

  ‘Sounds like we did the same.’ I looked back at the lawn.

  The match ended and the winner raised his arms to punch the air with victory. The crowd made space for him as he began a lap. He made strong-man poses along the way, red-faced and panting. Muscles bulged through his tee-shirt.

  ‘What an idiot,’ I said. ‘Who is that guy?’

  ‘That’s Felix Stern.’ Sam laughed. ‘He’s your manager.’

  I knocked on Felix’s door and went in. ‘Hi, Felix.’ I sat in the easy chair opposite his desk.

  ‘Hi.’ Felix leaned forward. His head was small compared to the rest of him and he had a youthful face aged only by a worried frown, like a knot over his eyes.

  ‘So why isn’t METX working? Let’s make this quick.’

  I took a deep breath. ‘Most of these type of delays are network issues. They’re taking a look and in the meantime I’m running tests on my code.’

  ‘Look at this list of complaints.’ He dropped a collection of printouts on the desk. ‘What are you doing to make this list go to zero?’

  I flicked through the pages. I already knew about the complaints.

  ‘I’m trying to have the issue investigated from all angles,’ I said clutching the paper. ‘The server seems to be under a strain but I’m not sure if-’

  ‘Status,’ Felix read from my online update. ‘Server under strain... Not sure if this is usual... Delays probably due to network problem... Alternatively a resource overload due to volume could be causing the delays... Analysis of difference between test and production environments underway to assess significance on above-mentioned... ’ His eyes flicked up and down the screen as if he was searching for more information.

  I sat forward to explain. ‘A delay can be caused by a number of issues. It might be a memory leak eating up resources. It might be the quality of the physical line. Or it might just be the feed’s volume is too large for this particular server.’

  ‘Why was this not caught during test?’

  ‘The test server is different to the live server. So that’s why this could be a server issue. Also, METX didn’t provide us with live data for our testing so we had to make do with sample data. So this could be a problem with the live data that simply didn’t exist before.’

  ‘Sample data? Why wasn’t the feed stopped from going live?’

  ‘Sorry?’ I stared at Felix, my brain racing. ‘I don’t understand.’

  ‘The Exchange refused to provide proper data for testing. What did you or Exchange group do about it?’

  ‘I don’t think they did anything.’

  ‘You don’t think they did anything? Is that a joke?’

  ‘No. I just didn’t talk to them about this specifically.’

  ‘What? What are you telling me now?’ His voice filled the room. ‘Your project had an inadequate test environment and you didn’t tell anyone about it?’

  ‘It was properly tested with all the available data,’ I pleaded. ‘A lot of feeds don’t have a proper test environment.’ I thought of other projects I knew in the same situation. ‘The business analysts used to handle the test side of things,’ I added quietly.

  ‘But that’s your job now. To push for what your feed needs.’ He poked the desk with a finger.

  I glanced around the room, looking for answers. There was a large window behind Felix. Light came through the half-open blinds in slices. Neat technical folders decorated the walls on either side of the room and on the floor in the corner a gym bag lay open with two metallic weights sticking out. My eyes went to the high-energy muesli bars on his desk. Next to them was a golf caddie pencil holder. It was empty but it had a photo leaning against it that I could only see from behind. Felix took a heavy breath and a logo expanded across his shirt. ‘CPR - The Kick of Life’.

  ‘What about analysis of the server before the feed was uploaded? Did you do anything about that?’

  ‘I had no reason to think there’d be a problem there,’ I admitted. ‘The live environment is allocated by Feed Support.’

  ‘But you’re supposed to check these things.’ He poked the desk again. ‘What have networks said? Could it be a bandwidth problem? A hardware problem?’

  ‘I don’t know,’ I squirmed in my seat. ‘They’re analysing the router configuration at our end and I expect they’ll also do a data trace to see if there’s a delay at a certain point. But it takes a while.’

  ‘You’ve got to keep on them.’

  ‘I will.’

  ‘Keep on their ass. What about the Exchange? Have you spoken to them?’

  ‘I sent them an email to check on their side.’

  �
��Ring them.’ He gave me a hard stare. ‘You’d better come up with a solution real quick. This is an important feed and it was given to you.’

  ‘I understand. I’ll keep you informed,’ I said.

  He went back to his keyboard and I took that as my cue to leave.

  An email was waiting for me from the Exchange when I got back to my desk. There was no obvious problem on their side. They hadn’t received any other complaints this morning.

  My phone rang. ‘Come in come in,’ I breathed, cradling the receiver.

  ‘You in the trenches, soldier?’ Gary asked.

  ‘Front line, Gary. What’s the word?’

  ‘Right,’ he sighed. I could imagine him flicking through the crumpled pages of his notebook. ‘I’ve been monitoring the data coming in from our router, yeah? From 8 a.m. London time onwards. I’m not seeing any delays.’

  ‘What about from the Exchange to their routers?’

  ‘It’s possible, but the overall turnaround is ninety milliseconds. We don’t expect faster than that.’

  ‘So, the packets from the Exchange are coming straight in to our live servers, consistent, no break?’

  ‘Yep.’

  ‘What about the dropped packets?’

  ‘The line isn’t dropping packets. Your feed is just rejecting them.’

  ‘Oh.’

  ‘The thing is, with the type of connection you have, if your feed doesn’t lift packets off the line quickly enough, it will just skip them, to alleviate congestion. That’s what I’m seeing here – Packet skipped due to congestion. I’m seeing another one now.’

  I swallowed hard. ‘So it looks like we can assume there’s no connectivity problem?’

  ‘That’s right.’

  ‘What about server performance?’

  ‘Between... what are they... your test server is T-A-4 and the METX live server is L-E-35. All libraries are the same. You’ve got all the same thread-handling. Same scheduling and architecture. There’s nothing at all to suggest a performance difference.’

  ‘Thanks a mill’ for checking, Gary.’

  ‘No worries, soldier.’

  ‘Over and out,’ I replied in our usual goodbye.

  ‘Fuck.’ I banged down the phone. With options closing in around me it was looking like an obscure mistake in my own code was the most likely blame. I ran the performance tests again, this time replacing the sample data with real data captured from the live feed. It was difficult to work with the larger volume but it meant the tests would be more accurate.

  Every negative result felt like a little reprieve but it was replaced with a fresh batch of nerves; I still had no idea what the problem was. Desperate, I began taking individual modules of code and testing them as isolated cases. I had to find something that was causing memory to bloat and slow down the feed. I noticed people coming back from lunch and realised no one had invited me. This was the first time I had been involved in a live problem. It felt like a landslide.

  Complaints and queries about progress were still popping up on my screen but I was running out of answers. Ignoring them I began trawling over my code. I mentally stepped through each algorithm calculating what it was doing and what could possibly go wrong.

  Boris was back at my cube. ‘We’ve got to get this fixed before New York opens and we’re flooded with US activity.’

  ‘New York opens in half an hour Boris. We’re not going to solve the problem, test it and roll it out in that time.’

  ‘Well, do you have any move on? Any update?’

  ‘It looks like my feed has some major memory issue slowing everything down.’ I dropped my head into my hands. ‘But my performance analysis isn’t showing any memory leaks.’

  ‘Let’s have a look,’ Boris said pulling up a chair.

  I showed him my test results, explaining what each module was doing and how the results showed normal behaviour. We stepped through the large algorithms together, just in case I had missed something.

  ‘How high is the memory usage now?’ Boris asked, out of ideas.

  I ran the command to check memory on the live server.

  ‘It’s down to five per cent.’ I narrowed my eyes.

  ‘What? That’s good, right?’

  I ran the command again ‘Back up to forty-five per cent. Not good.’ I licked panic off my lips and opened the log file looking for error messages. ‘Another skipped packet,’ I read. ‘Oh.’ I jumped and slapped my forehead. ‘The memory’s not too high, Boris. Every time there’s a dropped packet, there’s a fluctuation in memory.’ I felt sick thinking of the time I had wasted.

  ‘But why?’ Boris pressed.

  ‘Because METX requested we process the packets of data in sequence... So, when there’s a missing packet, the feed halts processing and shoots a message off to the Exchange server looking for it. Then the feed just sits and waits for it to arrive. But in the meantime, newer packets are still arriving and they have to be held on a queue, each one taking up memory.’ I shrugged helplessly. ‘Then the missing packet arrives, all the packets get processed and all the memory is released.’

  ‘Then why is your feed skipping packets in the first place? That’s the real question.’

  ‘No.’ I said firmly. ‘The question is correct. Why is my feed not performing fast enough? But, it’s got nothing to do with memory. The memory fluctuation is a symptom not the cause. I’ve been chasing a red herring.’

  A bell tolled and the US market opened. I stared through my fingers at the terminal imagining the New York trading beginning. Phones starting to jump. Orders and trades entering the system. And then while I was still holding my breath, the delays got worse.

  ‘Oh, fuck,’ I said through my hands.

  ‘Fuck, fuck,’ Boris echoed next to me. ‘Orla – what are you doing to me?’

  A new warning flashed on the screen.

  Tradeq currently experiencing delays

  ‘Whoa, what’s going on there?’ Boris grabbed the keyboard and punched in the Tradeq keys.

  We watched the prices updating with a drag.

  ‘Boris,’ I braced myself. ‘I think Tradeq and METX are running on the same server.’

  ‘Orla. Are you telling me your feed is actually compromising the server it’s running on?’

  ‘It’s possible.’ I pulled the keyboard back and checked the server statistics. The memory was high. I checked again. Still high. ‘It’s not going down.’

  ‘What’s not going down?’

  ‘Memory. It’s staying high. Performance is going to keep getting worse.’

  ‘Jesus, Orla, you’ve got to find the bug right now. Felix is going to kill us both.’

  A new round of irate messages began popping up obscuring the slow pricing and, overwhelmed with nerves, I began to giggle.

  ‘It’s not funny. What are you laughing at?’ Boris threw his hands at the screen.

  ‘I’m sorry,’ I choked, watching the flashing messages fighting each other for space.

  Felix was suddenly behind us making me jump.

  ‘I have traders yelling down the phone threatening to come up here,’ he said, pointing towards his office. ‘Two feeds are down. What the fuck is going on?’

  ‘I think METX is compromising the server it’s running on,’ I said, quickly sobering up. ‘I’ll move Tradeq to its backup server.’

  ‘Do it now,’ Felix said.

  With Felix and Boris standing behind me I restarted Tradeq on its backup, safely out of the way. The three of us stared at the screen until the Tradeq prices resorted to normal.

  ‘Why wasn’t this done before?’ Felix demanded.

  ‘I was trying to find the problem. I didn’t think the delays would get any worse.’ I squirmed, knowing the answer wasn’t good enough.

  ‘I take it METX is a software problem. What about stopping it and restarting it?’

  I hesitated. ‘On start-up it sends a message to the Exchange server requesting all its market data from the beginning of day. That would flood
the terminals with duplicates from this morning.’

  ‘Is the start time configurable?’ Boris asked.

  ‘Yes, 8 a.m. is the start-up time.’ I followed his train of thought. ‘If I just change it to 2 p.m. we’ll only get a small amount of duplicates.’

  ‘Okay.’ Boris said. ‘Start it on its backup server, wait until the duplicates are processed and then switch clients to the backup.’

  ‘Do it now,’ Felix barked.

  When I restarted METX on the backup the delays disappeared. When I was sure it was processing data correctly I re-directed traders to the backup providing them with timely data. The three of us watched as updates began to appear on the screen. I checked the performance.

  Memory 14%

  CPU 12%

  ‘This looks okay.’ Adrenalin began to drain from my body like blood. ‘I’ll keep an eye on it. I can restart the feed again if delays build up.’

  ‘I don’t want any more delays today,’ Felix said.

  ‘What the hell is going on?’ Boris asked, wiping his forehead when Felix was gone.

  ‘I think it’s dropping so many packets that it doesn’t have time to recover before the next packet is dropped,’ I said limply. ‘So the memory builds until it starts to use up memory that’s meant for other feeds, slowing down everything around it.’

  ‘Shit me,’ Boris said, wiping his brow again.

  ‘I’ll keep an eye on it, Boris,’ I assured him. ‘At least it’s isolated now so it can’t affect any other feed.’

  ‘Yeah, well, once the opening activity in the US is over things might quieten down. Alright, let me know if you need any help.’

  I shook my head at my code. The figures were dancing in front of my eyes. I turned back to the server stats again and watched the memory usage rise and fall along with the butterflies in my chest. Why aren’t you fast enough? I thought, feeling queasy. I realized I needed to eat.

  In the canteen I felt too sick to eat. I made my way to the juice bar where groups of brittle girls and sales boys posed like fruit flies. I avoided eye contact in case my realness was a shock to them and filled a paper cup with orange and mango juice.

  On the way back I stopped in to the Think Tank. The door whispered closed behind me and suddenly the pace of life was set by the fish traversing the wall to wall tanks. The only sound was from bubbles rising and the hum of a disillusioned grad playing with a yo-yo in the corner. Sam was in his usual spot, sitting against a giant lego block, leaning over a notepad on his knee. He was one of the few people who used the room for its actual purpose – somewhere to concentrate, away from the noise.