More Hygiene in IT: The Complete SAC Interview
Due to the skills shortage, more and more work is being outsourced to specialized IT service providers. As a result, the demands on these providers are constantly increasing. At the same time, the complexity of tasks is growing. Systems must be scalable, quality must be high, and processing times and costs should be kept as low as possible.
To optimally meet these complex requirements in the future, IT process automation was essential for SAC GmbH from Darmstadt. In the following interview, SAC Managing Director Thorsten Podzimek explains the diverse benefits of automation – from automated user management to 24/7 support and intelligent license management.
Mr. Podzimek, what exactly does SAC GmbH do?
The main part of our work is the complete IT operations of our customers. These are clients who sometimes have no IT employees of their own and don't want them either. We provide them with an all-round infrastructure service in the form of:
- 24/7 end-user support
- PC setup and installation
- User account creation
- Local server infrastructure setup
- Resource provision from our own data center
- Planning and project coordination
Additionally, we optionally take over the entire Office365 integration and administration. Everything except specialized applications is fully managed and operated by us, including telephony.
What does your customer base look like?
As part of these operations, we support mid-sized customers up to global players, for whom we serve as an "extended workbench" in various areas. In this context, the customer outsources various services to us. These customers have their own IT, but we partially handle the hosting and provide specific application or partial area support. We are broadly positioned but also extremely specialized. Everything we do, we can handle down to the lowest system level. Our policy is that we don't accept or do anything that we don't have expertise in.
Use Case 1: Automated User Management
Why and for what did you need an automation solution?
We found that we spend approximately 5,000 hours of work per year creating new employees for customers and deprovisioning (when employees leave the company again). It's clear that you need people to do this work. However, we always faced several challenges:
1. Discontinuity
This type of work doesn't come continuously, but always in bursts and time-bound. A typical scenario: A customer reports on Friday at 4 PM that four new employees are starting the following Monday and the corresponding accounts need to be created. Creating each account takes between three and four hours to set up the user in all IT systems. Specifically, a service employee at our company must work late into Friday evening. Testing with the user or customer is usually no longer possible at that time since no one is reachable. This can result in not everything working as desired on Monday or not all access being available. This creates rework that causes additional costs and often unnecessary discussions.
2. Fluctuating Quality
Depending on the person performing the task, this topic is very difficult to stabilize. The quality of execution often fluctuates depending on who does it and when it was performed, despite exact instructions and checklists.
3. Unspecific Information
Sometimes there is only inaccurate or incomplete customer information about the user to be created. It can happen that names are misspelled, titles are forgotten, or departments are still unknown. This additionally complicates an efficient IT-side onboarding process. In such cases, all information often has to be completely re-entered – a high effort for both sides.
What was your first approach to solving this?
To properly collect this customer information, we developed a Word form where you could fill in fields, had certain specifications, and fields were already preset (e.g., selectable distribution groups for the user). We made this document available to our customers. The customer filled it out, printed it, signed it, scanned it, and sent it back to us. We then filed it and processed it accordingly.
How practical were these forms?
In the long run, we couldn't keep up with the customer's dynamics in the forms (for example, when a new distribution group is added or removed). It would have been necessary to constantly adjust these documents, which would have meant immense effort. Moreover, every form change would have required an adjustment in handling on the customer's side. This approach was therefore impractical and inefficient in both directions.
What requirements needed to be implemented?
We needed a solution that included the following 3 points:
- A dynamic form that adapts to the customer's current situation and queries current conditions. This was to ensure that form maintenance is minimized, we have correct input, and can ensure that the entire process can be completed without errors.
- Completely automated creation of user accounts including all related information.
- The entire communication including approval processes and questions.
We solved this with automation. We developed an onboarding workflow for our customers that not only creates the user but also handles everything that needs to happen outside of IT. Today, we can automatically set the following key data in the customer portal:
- New employee starts on date X
- An appointment for onboarding training is set in Outlook
- There are corresponding onboarding tasks in Planner
- The business card manufacturer receives an order directly (because the data is already available)
- The building manager receives an email to prepare a nameplate for the door
- All actions have a corresponding completion note
And what about offboarding?
The onboarding process is already complex, but when an employee leaves the company, it becomes even more complex. In this case, their access permissions must be revoked on a scheduled basis. Other processes that should only be triggered after a certain time or at a specific time are also part of this. In other words: if an employee starts next Monday, the account can be created immediately. But if the employee leaves at the end of the month, the accounts should only be locked at that month-end date. However, this can be entered in advance on a scheduled basis and thus also automated.
What advantage do the automated processes provide for you and your customers?
Automation offers the enormous advantage that tasks can be activated on schedule and downstream processes can be mapped simultaneously. We catch these important "hygiene measures" with automation. An example: An employee's email address or mailbox should remain for 3 months after departure and only then be deleted. Until then, the email is forwarded to a specific other address.
Due to the ever-increasing workload in IT operations, planning and follow-up of such information is often lost in day-to-day business. Our workflow "forgets" nothing and would still become active exactly on schedule 3 months later. Or it sends a reminder at a specific time asking whether the respective action should be finally executed. Some customers say "Ask me again in 3 months whether we still need the email."
Can you name other advantages?
Another advantage concerns security: When one of our service employees creates accounts for customers, they need administrative permissions there. This naturally also means a risk for the customer. Depending on the company size, there are questions from the compliance department about how we ensure that our employees don't steal data with their respective admin rights, accidentally delete anything, or keep access credentials securely stored. Through the workflow, it's possible for all administrative processes to be performed by the operator without admin rights. This provides us with an additional, major security feature.
Looking at the market in general, how do you differentiate yourself from other IT service providers through automation?
That was the context with which we started automation. We said: if we can achieve this, we have an enormous lead in the market. Currently, no one can offer this complexity and scope of services.
Automation forms the foundation for us to also implement the desired change in business model: away from the pure time-for-money model, towards managed services with flat rates for e.g., user and server packages. This makes the business model independent of IT service headcount and thus scalable. The economies of scale cannot be achieved without powerful automation software in the background.
The ability to scale is a very important topic in this context. If not four, but forty users need to be created simultaneously, you quickly reach the limits of what's feasible with manual processing. The speed and stability of an automated process are unbeatable here. The workflow creates the desired number of accounts completely in just a few minutes – without errors or rework. Manually, each individual account would mean up to three hours of effort alone.
As a Managing Director, you primarily have to think economically. What influence does automation have on your company from this perspective?
Automated processing is also highly attractive commercially. This allows us to compete in the market with a completely different price. Calculating 100 euros per hour, manually creating a single user would cost the customer 300-400 euros. Once the workflow is developed, there are practically no (variable) costs anymore. You have to install, license, operate, maintain, and service the automaton, but the direct initial costs are eliminated. No matter how many users I onboard or offboard – it doesn't cost me more skills or manpower. In the best case, the customer can self-administer and independently perform all user management. If needed, they can still call the service and get help, or delegate it to one of our service employees. That's the new world and truly a "wave-breaker job."
Just through the automation of 5,000 admin hours per year, there is a saving of approximately 2.5 administrator positions per year. This doesn't mean these positions are eliminated. However, the corresponding employees have more time to focus on other tasks and new automation. Including the operating costs for the automation solution, we currently calculate savings of about 100,000 euros per year for this case alone.
Use Case 2: 24/7 Support & Incident Escalation Chain
Can you describe another scenario where you use automation for your customers?
On our automation journey, we have realized many more workflows and explored application areas. We have also invented services that we couldn't have implemented before without automation.
An example: We offer 24/7 service. In this context, the ratio of action and reaction is of immense importance. One of our customers is a large logistics company that operates around the clock. If their Oracle database fails at 3 AM or has poor performance values, logistics comes to a standstill. No goods leave the warehouse anymore and a large number of employees can no longer work or commission the goods. The customer has indicated that a failure can quickly be in the six-figure euro range if goods in and out no longer work.
How do you solve this problem at 3 AM on a Sunday night? First, an automatic root cause analysis must be performed as quickly as possible. For this, we have a monitoring system that triggers certain measures or further processes at certain threshold values. In the specific case: triggers the automation for the support escalation chain for troubleshooting.
What was the previous procedure in such a case?
In the past, the customer would call us at 3 AM, or rather a call center that was connected upstream. This service provider logged the failure in our system and called various on-call technicians one after another to determine who was available next. Then this on-call technician qualified what actually needed to be done and in turn called a 24/7 technician from an Oracle database specialist. This specialist then logged into the system and took the necessary steps to fix the error. This meant that the response time from the reported incident to handling by the right contact person was often an hour or more. Additionally, at least two people were busy recording the incident, qualifying it, and initiating the forwarding to the actual support engineer for troubleshooting.
How do you address this challenge today?
We have completely replaced this with a workflow. The exceeding of the respective threshold value is transmitted to the automation. Through a telephony integration, a list of on-call technicians administered in Flowster is called and the incident is reported. Via an additional SMS, the on-call technician is requested to confirm the alarm and thereby indicate that they are taking over the handling of the incident. This ensures that the calls don't go nowhere and someone actually takes care of it. The entire process is documented in an audit-proof manner through corresponding tickets and the respective status is also reported via Teams, email, and monitoring dashboard. Thus, everyone can see at any time what the processing status is and what was done by whom and when.
In summary: today we automate the detection, qualification, and reporting of such an incident. The monitoring detects an error, Flowster works through the right steps one after another with our configuration, a third party then does their work and provides feedback. Additionally, the entire process is also comprehensively documented and communicated. This would not have been possible before in this manner and especially at this speed. We also use this qualification and reporting procedure with other customers. At various data centers, for example, this works similarly regarding incident reporting via BMS systems.
Use Case 3: Automated License Management
Mr. Podzimek, automated user management already offers a variety of advantages for you and your customers. How can automation be used effectively in this context?
An interesting use case is license management. This is an underestimated cost factor in many companies. In the past, our customers often had a problem: when a new user is created, they need certain software license(s) to be able to work. If an employee leaves the company and the customer doesn't return the license, they still have to pay for it. Especially with regard to annually running licenses, it's economical to have a background process that also automatically returns unused licenses.
We have implemented the purchase and return of necessary licenses in an automated way. Our distributor developed an interface specifically for us for this purpose. Today we buy, for example, Microsoft licenses through the automation platform – and also automatically return them as part of the offboarding process.
Use Case 4: Certificate and License Monitoring
Large companies in particular have a vast number of licenses and certificates. It's easy to lose track of how long these are valid. Can automation also be used in this case?
The regular checking of certificates or licenses is another use case we developed for our customers. This means we automatically check whether and when licenses or certificates expire. This service is very popular with our customers. The workflow reports, for example, 4 weeks in advance that a website certificate is expiring. The customer then automatically receives a reminder email to object by deadline X. Otherwise, the certificate is automatically renewed or extended and the customer doesn't have to worry about anything. However, they would also have the option to say: Stop! I want to decommission the website or do something else.
Use Case 5: Automated Security Queries
You already mentioned security. Administrative processes can be performed by you without admin rights at your customers. How can your customers use automation to optimize their security?
Another use case concerns automated security queries. One of our customers checks daily whether their company's email addresses and all telephone and mobile numbers appear in hacker lists or are being misused. This normally only happens when you receive suspicious emails or have an initial suspicion. You could manually enter the addresses and numbers on a website and test them as needed. We have automated this.
Through AD, we have access to customer data. This means the workflow checks all AD objects on the internet for suspicious email addresses. If suspicious, it issues a warning message, combined with the question of what to do next: "Address XY has been on the 'Dropbox' list since yesterday. What should be done?" Then you know you need to change your password at Dropbox. Or the password was stolen on social media. You normally only find out about this when something has already happened. This is a small, optional service that the customer then subscribes to monthly. The workflow runs automatically once every night. We have a long list of this type of workflows, which we call "IT housekeeping," and we implement them successively.
Use Case 6: Automated Updates with Rollback
Finally: Is there another use case that you think is particularly interesting for IT service providers?
Automatic installation of updates is a huge topic. We also do this for our customers in an automated way – including backups, rollback, and functionality testing. Installing Windows updates isn't complicated, but the important question is: does everything still work afterwards? The entire process surrounding it is much more complex than the updates themselves.
The customer wants, for example, that the downtime is announced with appropriate advance notice. In the case of an update process, we also check whether users are still logged in (on terminal servers). These are automatically informed to finish their work and after an appropriate time are also automatically logged off. However, the process can also be aborted by defined responsible persons in such cases. The automation allows interaction here to possibly not start the execution, since the systems are still needed.
In the further course of the update process, the workflow checks whether a current backup is available and thus the possibility for a rollback is given. Subsequently, a snapshot is automatically created if necessary, the updates are performed, and the system is checked for functionality following the update. If the update was not successful, an escalation message is automatically sent to the on-call team. They check again or decide on an automatic rollback. The snapshot is resolved again and a final message is sent (updates successfully performed or not).
This is the work of people today who have to do it in the evening or on weekends. Therefore, mistakes can sometimes happen – for example, forgetting to inform colleagues about the upcoming update or to turn off a monitoring system. All of this doesn't happen with automation.
Thank you very much for these fascinating insights, Mr. Podzimek!
Would you like to achieve similar successes in your company?
Learn more about the partnership between FLOWSTER and SAC in our detailed user report.
Or contact us directly for a non-binding consultation.