From Competitor to Captain: My CCDC Journey

This article documents my personal experiences with CCDC. The opinions expressed are my own.

I’ve had the privilege of participating four times in the Collegiate Cyber Defense Competition across five years. During my time as a competitor, I learned volumes about system administration and working together with a team. My primary role was Linux administration and hardening, but other roles rubbed off along the way giving me a broad view of the competition.

If you are not too familiar with CCDC, here is the overview: CCDC is a blue-team network and business simulation run in the United States, which fills a distinct niche compared to more traditional security competitions like CTFs by focusing on network fundamentals, service management, incident detection & response, and communication. Universities from around the nation compete in this yearly event with the hope of winning the National title. The gist of the competition is as follows: you are a newly hired IT team tasked to bring a business’ network into a secure and known state as quickly as possible. The catch is that you have multiple bosses breathing down your neck, assigning you business tasks (known as injects) quicker than you can complete and an active red-team (professional penetration testers) breaking down the doors of your network and trying to keep your services down. Teams are thrust into this overflow of tasks and events and scored on their performance. Although it may sound like volunteering to become a piñata, playing in CCDC is some of the most stressful fun you can have. You have to experience it for yourself to truly understand the continual adrenaline rush of jumping into an unknown, broken, red-team infested, network of machines.

Sound fun? Great! So how do you get started? In my case, my undergraduate university, the University of Central Florida, started its own security club, Hack@UCF, also known as the Collegiate Cyber Defense Club on paper. The club’s primary directive was to form a team to play in CCDC. I joined the club immediately given my interest in cyber security and I decided to give CCDC a go. This turned out to be one of the best decisions of my undergraduate career.

After playing for two seasons, I left UCF with extensive CCDC experience and joined the University of Florida to continue my education as a Ph.D Graduate Student. UF has a decade-old security club known as the UF Student InfoSec Team or UFSIT for short. What UFSIT did not have, however, was a CCDC team. In this, I saw an opportunity to bring my knowledge and experience to a new school. In early January of 2016, I decided to found the inaugural UFCCDC team. I realized once I started training UF’s team that I was not as familiar with many of the subtle details that team members in other roles such as captain, business, or Windows admin knew. This was a side-effect of the excellent division of labor on UCF’s team, but it made starting UFCCDC more difficult as I had to recover state and strategies from the recesses of my mind. As such, I believe others may find my thoughts and ideas on starting a CCDC team from scratch useful. I cannot cover every detail of my experiences, but I hope to capture my transition from competitor to captain.

Figure: A timeline of my participation in CCDC across two schools over the last six years.

Starting Fresh at UF: Finding a Team Sponsor

Step one of my change from competitor to captain at UF was getting support for my effort, starting with a sponsor. Before starting a CCDC team at a university, you will need a faculty sponsor. They are responsible for the administrative aspects of the competition (registration fees, transport, getting excused absences, travel, etc.). In some cases, this sponsor may also double as a coach. At UCF, Dr. Tom Nedorost sponsored the team, along with Hack@UCF. He was instrumental in driving up departmental support, fighting for funding, and managing many other team logistics, letting us focus on the competition.

Since I had just joined UF as a grad student, I was less familiar with the department, but I happened to know Dr. Joe Wilson from my preview visit. I learned that Dr. Wilson was always current on industry security tools & techniques. Combined with his malware reverse engineering and penetration testing courses and that he was sponsoring UFSIT already, having him sponsor the CCDC team was a no brainer.

Finding Members

Once you have some sort of faculty support, you need to start finding applicable students. If you are lucky, your school already has a cyber security club. At UCF, the club was started with the primary purpose of playing in CCDC (hence, its official name). At UF we already had UFSIT, which made my job a lot easier as I was already attending SIT’s meetings regularly.

A key step before thinking about creating a tight-knit team is to gauge interest across as many relevant clubs and departments. I recommend creating a Google Form that captures key information (name, email, experience) and distributing it regularly. Having a larger applicant pool is better than having too few qualified members. In my case, I limited my search to UFSIT and the immediate Computer Science department, which was convenient. Depending on your university, you may want to reach anyone in engineering or even business as CCDC does involve a lot of technical and non-technical writing. If your university happens to have an Information Technology (IT) degree, in my opinion, this would be the best place to find interested and qualified students. Another really good trick if you have the sponsor support is to create a “CCDC class”. This is typically a fully-fledged class but with the behind-the-scenes purpose of blocking off time for students to train. This ensures that new and returning members have a guaranteed ability to attend trainings and lectures and pure training can be separated into two timeslots. UCF employs this to great effect and Dr. Wilson had an Enterprise Defense course which fed nicely into CCDC (and got us one of our first team members).

Our first year (2015-16) of recruiting at UF was a bit haphazard and last minute as I wasn’t quite sure I was up for the task of leading a team two months before the qualifier, but we managed to find around 15 interested students. The next year (2016-17), over half of our team was returning and our applicant pool was about 5 students larger. The key takeaway is when starting a team have a well-defined pipeline for students and don’t let it stall out. Over subscribe your applicant pool to increase your chances of finding the right mix of students. Students that don’t make it to the final team roster should be kept in-the-loop with team activities.

UFCCDC Training

Once I had a pool of applicants, we started to train. I prepared various lectures on topics such as:

• Networking basics (TCP/IP, subneting, application protocols, etc.)
• Linux Tricks (GRUB bypass, command line fu, important tools)
• Linux Hardening (iptables, /bin/false in /etc/passwd)
• Windows Active Directory
• Windows Group Policy
• And more…

The training fell completely to me during the 2016 season as the entire team, except me, was green. This was tough to manage while starting my Ph.D as it took about 15 hours a week. Not to mention I wasn’t familiar with all of the topics that needed to be taught (e.g., windows administration). Luckily some members had practical experience with some of these topics already and could help other members debug issues. The single most time-consuming part was preparing for these mini-lectures. Creating slides and preparing VMs took many hours, plus then we needed to sit there and train with. This only made me appreciate more how well oiled the UCF training and practicing was when I was a member.

To train, we sat in a computer lab with remote access to a well-provisioned VMWare ESXi (generously provided by our faculty sponsor) instance for us to spin up and configure VMs as we pleased. The members would follow along by watching my VM’s console through VSphere as I demonstrated the key topics. Then we’d break and everyone would try and replicate the steps on their own VMs. We met once a week for about two to three hours and I took attendance at each meeting. Attendance later helped narrow down 12 members for the official CCDC roster. A major key to being able to practice was VMWare. Having the ability to snapshot and spin up different operating systems for practice was essential to getting more familiar with hardening and securely configuring various systems. Plus, the regional and national competitions expect you to be familiar with ESX, including defending it from attack or weak defaults. If your department doesn’t have access to these types of resources, talk to your faculty sponsor about getting them or a close equivalent.

Not surprisingly, there is a direct correlation to training time to competition performance. For instance, UCF last I checked, was meeting three days a week many months out from the regional competition to get prepared. This is definitely on the upper end of practice time from what I’ve heard, but practicing working together as a team and learning how to respond to simulated events with deadlines and stress closely mirrors the actual competition. Technical skills can only get you so far – at the end of the day you need to be comfortable with your team members and their skills.

Selecting a Team Roster

After about two months of training, I was getting to know the regular members and trying to assign them to specialties. I knew we needed two to three people working on Windows administration, two to three people focusing on Linux, a business manager, a networking / firewall person, a captain, and a co-captain. Some people will have a dual-role, but it was critical at we had at least one very strong person focusing on Linux and another on Windows. For our competition strategy, I ended up as the primary Linux administrator instead of the captain during the competition. We knew that the captain would be “busy” periodically during the competition, which would affect my time on a machine. In my opinion, your captain shouldn’t be your most technical person. If you have a choice between seniors or sophomores, make sure that you understand that senior students won’t be there next season (unless they delay their graduation because they love CCDC so much, which I’ve seen happen). An all-senior team is a team that won’t exist next year.

If you’re lucky and you have more applicants than slots on your team, making the final team roster can be a little awkward. I recommend that you let your faculty sponsor make any tough decisions, even if by proxy, to avoid any hurt feelings. Also collecting data during trainings can help justify decisions if there is any conflict. In my case, this was never an issue, but keep it in mind as your team grows.

The Qualifier

Note that some parts from here on out will be SECCDC specific. I advise that you search for information and talk to other teams specific to your region.

After months of training and team building, the time to play will come. First on the CCDC bracket is the qualifying competition. This is per region and all interested schools will apply and submit their fees. This stage of the CCDC bracket is completely remote and lasts only four hours. It’s quite easy compared to the regionals, so don’t expect that getting first place at this stage to guarantee your victory. Given my past experience with the CCDC qualifiers I was able to prepare the 2016 and 2017 UF CCDC teams well for what to expect from the SECCDQC.

In recent years, they let you preview the environment for a day to examine the topology and security state of machines. This didn’t use to be the case. In fact my first qualifier with UCF required us to have a physical ESXi server onsite and to top it off, southeast shipped us the competition images via BitTorrent! Needless to say there was no red team, just injects and service scoring. Today the qualifiers are remotely hosted by Moraine Valley using NETLAB, a GUI for ESXi offering remote KVM and power state management of a virtual network.

Figures: Moraine Valley’s remote NETLAB GUI and network topology for SECCDQC 2019

Enjoy the preview access it while it lasts and take excellent notes while looking for pre-seeded vulnerabilities. Your team can be better prepared for finding these types of misconfigurations if you have practiced with your own insecure images beforehand.

The competition itself just requires a room to be reserved and an external judge to be present. In our case, we found our external judge, a friend of Dr. Wilson’s, but not affiliated with UF, and booked The Dungeon – a particularly chilly computer lab where we had been practicing before. The qualifying event is short and intense with barely time for breaks. Injects flood in and services take hits from the red-team very early on. Lasting only four hours, the quals will be over before you know it. There is a lot less going on compared to the regional or national level, so use this as your team smoke test. You should come out of the qualifier with a good idea which areas you still need to practice before the regional.

Regionals

If you did well and were one of the top eight teams in your region, congratulations! You now get to experience the real competition. Depending on how far or well sponsored you are, this will be the time to figure out your travel plans and to make any adjustments to your roster, if needed. In my experience, if you are a fresh team, there is no substitute for actually playing in a regional competition. This is a bit of a catch-22 considering you need to be in the top eight, but if you can make it, soak in all of the details of the region you’re playing in. For example, our region, the southeast, has a reputation for lots of paperwork, so it makes sense to drill heavily on written injects and change control. Each region implements their version of CCDC from National’s guidelines, so each will have its own flavor, but the same overall scoring and organizational framework. Regionals is the real-deal. It has sponsors, lots of volunteer staff, and a professional red-team. It’s two full days long and has quite the amount of content from injects to required presentations to red-team incursions.

To help us prepare, SECCDC releases the network topology, which we recreated in our ESXi environment. Now we were able to drill on managing and hardening the expected services with snapshot support for quick resets. To help simulate active attackers better, we had a few of our members play the red team and gave them the initial password sheet for our network to start. This helped increase the stress and allow them to operate and deny service to less hardened services. If your school has a Collegiate Penetration Testing Competition (CPTC) team, those members would fit great.

Figure: The 2019 SECCDC topology as an example.

With our transportation secured, password lists printed out, and reference materials collated, we headed the Kennesaw State University (KSU) to compete. Our first year competing was quite an experience. We ultimately didn’t place our first year at regionals due to a snafu regarding our ESXi machine’s power state, but our second year we came back to place third. After I retired as captain to focus on my research and passed the torch over in 2018 to Ben Simon, an impressively skilled freshman, we placed in second!

Photo: UF CCDC’s 2017 team.

Nationals

Take regionals, multiply roughly everything except paperwork by two, and you have nationals. Twice the machines, twice the red team, and twice the fun. Making it to nationals is no small feat as you must have beat out over 100 universities to make it. Preparing for it is very similar to regionals, given all regionals use NCCDC as a model. The major differences are that there are truly more machines than physical team members and the red team play really dirty, even going as far as to develop custom stealthy malware each year. Beyond practicing harder than ever, another thing to consider as a captain is that the national competition typically falls towards the end of the spring semester, which means it may impact final exams. If it does, your sponsor can secure you official excusal forms to fend off any dismissive professors for your team. Despite this, when at UCF one of my professors would not let me make up an exam and instead dropped it despite my protest and my need for more points (*shakes fist*), so your mileage may vary. If you are really lucky your department chair will be onboard with the CCDC team representing the department, which could help persuade more stubborn faculty. The takeaway is that if you are one of the lucky teams to make it to nationals, expect your studies to take a back seat.

Unfortunately UFCCDC didn’t advance to the national competition during my tenure, although we came so close to beating my alma mater at regionals in 2018 😉. What I do remember from my time as a competitor on UCF’s team is that our first national competition we were woefully unprepared. We had the SECCDC model of play figured out, but nationals introduced many different elements, including more services, less practiced OS’ (Solaris, FreeBSD, etc.), and a significantly stronger red team. This combo knocked us about and sank our morale. The upshot is that next year we practiced even harder and came back to win the national competition in 2014. Like regionals, knowing what to expect from the competition ended up helping us more than expected. I took a hiatus after our first national win and White House tour, but UCF went on to utterly crush the NCCDC for 2015 and 2016.

The Second Year and Beyond

The first year of a CCDC team is likely the hardest. Everything needs to be started from scratch - team building, training, scheduling, and more. Some days can be a grind while others a blur. Coming out the other end of your first competition season, you realize just how much you’ve already learned. Take stock of what went right and what went wrong. Write down as much as you can remember about the events: inject descriptions, red team tricks, shell one-liners – write it all down. It doesn’t have to be neat but needs to be documented for next year, at least to jog your memory. Fine details will fade away even six months later. The key to coming back even stronger next year is adjusting your training to address your deficits. Document the VMs you created (descriptions, passwords, services) because the last thing you want is to have to drop into single-user mode for your network at the start of next season. If you have instrumental teammates who are graduating, plan to find a replacement within or outside of the team for next year. Start planning now for next year and have a timeline for the following fall. Once summer comes around, people will disappear, so set realistic goals. Don’t burn yourself out.

Closing Thoughts

Starting or just playing with a CCDC team is a big commitment, but the curricular and extra-curricular benefits to you, your club, and your university make it worth it. After helping the team off the ground for two years, I officially hung up my hat and handed the reigns of team training to the incredibly capable Ben Simon. My career as a CCDC competitor is over, but the memories, friends, and knowledge I built up during my time as a competitor and then captain will always be with me. Personally, I can’t help but feel a sense of “I built that” pride. I’m glad to have made a positive impact on UF’s security club. While I officially retired, I continued on with UFSIT as a graduate consultant, eventually organizing our first and second annual SwampCTF competitions. If you have any questions or comments, feel free to leave them below or DM me on Twitter. Thanks for reading!