SlideShare a Scribd company logo
Network Security in
Embedded devices
Serhii Korniienko - Embedded QA engineer
Examples of properties of typical
embedded computers when compared
with general-purpose counterparts are:
• low power consumption
• small size
• rugged operating ranges
• low per-unit cost
• often ‘non-standard‘ solution
Why low-level network testing is a
must in embedded:
• Often custom protocols used to reduce
CPU/RAM/ROM/bandwidth usage
• Low cost
• Timing properties of the program depend heavily
on its environment
• Environment =
+Process or & Memory Hierarchy
+ Operating System, other processes/threads, …
+ Network
+ I/O Devices
+…
Our experience:
1. Requirements
2. Approach
3. Tools
4. Test setup
5. Logs gathering and
analysis
6. Issues
7. Test cases
8. Process
1. Our requirements:
• Low cost – cheapest HW components approved by
customer (BoM < 20$)
• Http and custom UDP support
• Stack: freeRTOS + LwIP
• Rigid requirements on synchronization jitter (PTP)
• IC network switch to build daisy-chained network
structure
• Stable work in heavy loaded network (1Gb
wireshark dump file per minute)
2. Low –level testing approach:
• List of all low-level network stuff:
– Ports inventory:
– 80/tcp – http
– 554/tcp - rtsp
– 67/udp - dhcps
– 68/udp - dhcpc
– 319/udp - ptp-event
– 320/udp - ptp-general
– 2467/udp - custom protocol
– 17100/udp - custom protocol
L4 protocols inventory:
•ICMP
•UDP
•TCP
•IGMP
3. Tools:
1. NMAP – enumeration and packet sending
- sS
- sU
- sO
- sX
- sN
- sF
- sX
- sA
- sY
- --scanflags
3. Tools:
2. Ostinato –packet editing and traffic generation
- Set a value for any field of any protocol
- Open and edit PCAP files, replay and save back
- Support for the most common standard protocols
- Free, Libre, Open-Source Software
3. Tools:
3. LOIC – open-source network stress testing
and denial-of-service attack application
- IC Switch throughput test
4. Test setup
5. Logs gathering and analysis:
The key thing is to gather and analyze all available info from all
system components (switch and microcontroller) to recognize
and eliminate bottlenecks and issues:
- Buffer overflows and underflows
- Queue overflows and underflows
- Protocol/link state changes
- Tx/Rx error counters
- Switch counters
- Error messages
- Power dropdown
- Unauthorized recording
- Voltage levels
- …
6. Issues found:
Issue Web server goes
down during polite
TCP scan
Discovered/
Investigated
NMAP scan
Root cause LwIP configuration issue
Issues found:
Issue Discovered/
investigated
Root cause
Hard-fault occurs if send
fragmented UDP-packet
NMAP/
Wireshark/
Ostinato
LwIP incorrectly reassemble
fragmented UDP packets
Issues found:
Issue Discovered/
investigated
Root
cause
Device can be shutdown by specially crafted
control message
Script/fuzzing Input
validation
issue
Device can turn off it’s power supply and network chip power supply ☺
Issues found:
Issue Device hard-fault if send
abnormally large or small
message on control port -
Discovered/
investigated
LOIC/
Script/fuzzing
Root cause Input validation issue
Issues found:
Issue Discovered/
investigated
Root cause
Sending short ptp packet cause ptp
recalibration and packet loss
NMAP scan
Ostinato
LwIP issue
Issues found:
Issue Discovered/
investigated
Root cause
Network link goes down and Up
during nmap scan
NMAP/
Wireshark/
Ostinato
PSU issue for new IC
switch
Network link goes down during
nmap scan
NMAP/
Wireshark/
Ostinato
unauthorized command
on switch management
interface
Issues found:
Issue Discovered/
investigated
Root
cause
Device become unreachable if send short UDP
packet on port 319/320
NMAP scan
Ostinato
IC switch
issue
All devices in chain become unavailable if perform
UDP flood on port 319/320 of last device in chain
LOIC/Ostinato IC switch
issue
7. Test cases:
Fuzzing!
Test cases:
Test cases:
8. Process:
• Perform low-level testing to
every hardware changes
• Preform low-level testing for
every configuration changes
• Preform low-level testing for
every low-level structures
changes
• Preform low-level testing
periodically
Conclusion:
If you still hesitate about
the necessity to perform
low-level network testing
– go to the issues found
chapter!
Ad

More Related Content

What's hot (20)

OpManager training - Device discovery and classification.
OpManager training - Device discovery and classification.OpManager training - Device discovery and classification.
OpManager training - Device discovery and classification.
ManageEngine, Zoho Corporation
 
Monitor and manage everything Cisco using OpManager
Monitor and manage everything Cisco using OpManagerMonitor and manage everything Cisco using OpManager
Monitor and manage everything Cisco using OpManager
ManageEngine
 
Microsoft Infrastructure Monitoring using OpManager
Microsoft Infrastructure Monitoring using OpManagerMicrosoft Infrastructure Monitoring using OpManager
Microsoft Infrastructure Monitoring using OpManager
ManageEngine
 
VMware monitoring must haves
VMware monitoring must havesVMware monitoring must haves
VMware monitoring must haves
ManageEngine
 
Network Maps & Reporting [Free OpManager Training - Part 5]
Network Maps & Reporting [Free OpManager Training - Part 5]Network Maps & Reporting [Free OpManager Training - Part 5]
Network Maps & Reporting [Free OpManager Training - Part 5]
ManageEngine, Zoho Corporation
 
Network fault management and IT automation training
Network fault management and IT automation trainingNetwork fault management and IT automation training
Network fault management and IT automation training
ManageEngine, Zoho Corporation
 
Free OpManager training_Part 1- Discovery & classification
Free OpManager training_Part 1- Discovery & classificationFree OpManager training_Part 1- Discovery & classification
Free OpManager training_Part 1- Discovery & classification
ManageEngine, Zoho Corporation
 
Season 4 [Free OpManager training] Part1- Discovery and classification
Season 4 [Free OpManager training] Part1- Discovery and classificationSeason 4 [Free OpManager training] Part1- Discovery and classification
Season 4 [Free OpManager training] Part1- Discovery and classification
ManageEngine, Zoho Corporation
 
Proof of Concept Guide for ManageEngine OpManager
Proof of Concept Guide for ManageEngine OpManagerProof of Concept Guide for ManageEngine OpManager
Proof of Concept Guide for ManageEngine OpManager
ManageEngine, Zoho Corporation
 
[Season - 3 Free OpManager Training] Monitoring Server Performance
[Season - 3 Free OpManager Training] Monitoring Server Performance[Season - 3 Free OpManager Training] Monitoring Server Performance
[Season - 3 Free OpManager Training] Monitoring Server Performance
ManageEngine, Zoho Corporation
 
Free OpManager training Part 2 Monitoring Server Performance- season#3
Free OpManager training Part 2 Monitoring Server Performance- season#3Free OpManager training Part 2 Monitoring Server Performance- season#3
Free OpManager training Part 2 Monitoring Server Performance- season#3
ManageEngine, Zoho Corporation
 
Chapter08
Chapter08Chapter08
Chapter08
Muhammad Ahad
 
Chapter12
Chapter12Chapter12
Chapter12
Muhammad Ahad
 
Free OpManager training Part1- Discovery and classification season#3
Free OpManager training Part1- Discovery and classification season#3Free OpManager training Part1- Discovery and classification season#3
Free OpManager training Part1- Discovery and classification season#3
ManageEngine, Zoho Corporation
 
Gulf Chemicals & Metallurgy manages 1700 interfaces with OpManager
Gulf Chemicals & Metallurgy manages 1700 interfaces with OpManagerGulf Chemicals & Metallurgy manages 1700 interfaces with OpManager
Gulf Chemicals & Metallurgy manages 1700 interfaces with OpManager
ManageEngine, Zoho Corporation
 
OpManager Major Features
OpManager Major FeaturesOpManager Major Features
OpManager Major Features
tecanody
 
Season 3 [free OpManager training]_Part1- Discovery and classification
Season 3 [free OpManager training]_Part1- Discovery and classificationSeason 3 [free OpManager training]_Part1- Discovery and classification
Season 3 [free OpManager training]_Part1- Discovery and classification
ManageEngine, Zoho Corporation
 
Free Netflow analyzer training - diagnosing_and_troubleshooting
Free Netflow analyzer  training - diagnosing_and_troubleshootingFree Netflow analyzer  training - diagnosing_and_troubleshooting
Free Netflow analyzer training - diagnosing_and_troubleshooting
ManageEngine, Zoho Corporation
 
Free OpManager training Part 3 - Monitoring Network Performance and Network Maps
Free OpManager training Part 3 - Monitoring Network Performance and Network MapsFree OpManager training Part 3 - Monitoring Network Performance and Network Maps
Free OpManager training Part 3 - Monitoring Network Performance and Network Maps
ManageEngine, Zoho Corporation
 
Understanding firewall-policies-their-effectiveness-in-defending-against-netw...
Understanding firewall-policies-their-effectiveness-in-defending-against-netw...Understanding firewall-policies-their-effectiveness-in-defending-against-netw...
Understanding firewall-policies-their-effectiveness-in-defending-against-netw...
ManageEngine, Zoho Corporation
 
OpManager training - Device discovery and classification.
OpManager training - Device discovery and classification.OpManager training - Device discovery and classification.
OpManager training - Device discovery and classification.
ManageEngine, Zoho Corporation
 
Monitor and manage everything Cisco using OpManager
Monitor and manage everything Cisco using OpManagerMonitor and manage everything Cisco using OpManager
Monitor and manage everything Cisco using OpManager
ManageEngine
 
Microsoft Infrastructure Monitoring using OpManager
Microsoft Infrastructure Monitoring using OpManagerMicrosoft Infrastructure Monitoring using OpManager
Microsoft Infrastructure Monitoring using OpManager
ManageEngine
 
VMware monitoring must haves
VMware monitoring must havesVMware monitoring must haves
VMware monitoring must haves
ManageEngine
 
Network Maps & Reporting [Free OpManager Training - Part 5]
Network Maps & Reporting [Free OpManager Training - Part 5]Network Maps & Reporting [Free OpManager Training - Part 5]
Network Maps & Reporting [Free OpManager Training - Part 5]
ManageEngine, Zoho Corporation
 
Free OpManager training_Part 1- Discovery & classification
Free OpManager training_Part 1- Discovery & classificationFree OpManager training_Part 1- Discovery & classification
Free OpManager training_Part 1- Discovery & classification
ManageEngine, Zoho Corporation
 
Season 4 [Free OpManager training] Part1- Discovery and classification
Season 4 [Free OpManager training] Part1- Discovery and classificationSeason 4 [Free OpManager training] Part1- Discovery and classification
Season 4 [Free OpManager training] Part1- Discovery and classification
ManageEngine, Zoho Corporation
 
[Season - 3 Free OpManager Training] Monitoring Server Performance
[Season - 3 Free OpManager Training] Monitoring Server Performance[Season - 3 Free OpManager Training] Monitoring Server Performance
[Season - 3 Free OpManager Training] Monitoring Server Performance
ManageEngine, Zoho Corporation
 
Free OpManager training Part 2 Monitoring Server Performance- season#3
Free OpManager training Part 2 Monitoring Server Performance- season#3Free OpManager training Part 2 Monitoring Server Performance- season#3
Free OpManager training Part 2 Monitoring Server Performance- season#3
ManageEngine, Zoho Corporation
 
Free OpManager training Part1- Discovery and classification season#3
Free OpManager training Part1- Discovery and classification season#3Free OpManager training Part1- Discovery and classification season#3
Free OpManager training Part1- Discovery and classification season#3
ManageEngine, Zoho Corporation
 
Gulf Chemicals & Metallurgy manages 1700 interfaces with OpManager
Gulf Chemicals & Metallurgy manages 1700 interfaces with OpManagerGulf Chemicals & Metallurgy manages 1700 interfaces with OpManager
Gulf Chemicals & Metallurgy manages 1700 interfaces with OpManager
ManageEngine, Zoho Corporation
 
OpManager Major Features
OpManager Major FeaturesOpManager Major Features
OpManager Major Features
tecanody
 
Season 3 [free OpManager training]_Part1- Discovery and classification
Season 3 [free OpManager training]_Part1- Discovery and classificationSeason 3 [free OpManager training]_Part1- Discovery and classification
Season 3 [free OpManager training]_Part1- Discovery and classification
ManageEngine, Zoho Corporation
 
Free Netflow analyzer training - diagnosing_and_troubleshooting
Free Netflow analyzer  training - diagnosing_and_troubleshootingFree Netflow analyzer  training - diagnosing_and_troubleshooting
Free Netflow analyzer training - diagnosing_and_troubleshooting
ManageEngine, Zoho Corporation
 
Free OpManager training Part 3 - Monitoring Network Performance and Network Maps
Free OpManager training Part 3 - Monitoring Network Performance and Network MapsFree OpManager training Part 3 - Monitoring Network Performance and Network Maps
Free OpManager training Part 3 - Monitoring Network Performance and Network Maps
ManageEngine, Zoho Corporation
 
Understanding firewall-policies-their-effectiveness-in-defending-against-netw...
Understanding firewall-policies-their-effectiveness-in-defending-against-netw...Understanding firewall-policies-their-effectiveness-in-defending-against-netw...
Understanding firewall-policies-their-effectiveness-in-defending-against-netw...
ManageEngine, Zoho Corporation
 

Similar to Embedded Webinar #17 "Low-level Network Testing in Embedded Devices Development" (20)

Opmanager Workshop - Middle East
Opmanager Workshop - Middle EastOpmanager Workshop - Middle East
Opmanager Workshop - Middle East
ManageEngine, Zoho Corporation
 
Bell4GLTE
Bell4GLTEBell4GLTE
Bell4GLTE
Ishuranga Wijesinghe
 
New Generation Oracle RAC Performance
New Generation Oracle RAC PerformanceNew Generation Oracle RAC Performance
New Generation Oracle RAC Performance
Anil Nair
 
Packet Analysis - Course Technology Computing Conference
Packet Analysis - Course Technology Computing ConferencePacket Analysis - Course Technology Computing Conference
Packet Analysis - Course Technology Computing Conference
Cengage Learning
 
Industrial Ethernet, Part 2: Case Studies
Industrial Ethernet,Part 2: Case StudiesIndustrial Ethernet,Part 2: Case Studies
Industrial Ethernet, Part 2: Case Studies
ControlEng
 
Basic network training2
Basic network training2Basic network training2
Basic network training2
Arunchai Seangparch
 
6 profiling tools
6 profiling tools6 profiling tools
6 profiling tools
videos
 
Realtime traffic analyser
Realtime traffic analyserRealtime traffic analyser
Realtime traffic analyser
Alex Moskvin
 
Tech 2 Tech: Network performance
Tech 2 Tech: Network performanceTech 2 Tech: Network performance
Tech 2 Tech: Network performance
Jisc
 
Network State Awareness & Troubleshooting
Network State Awareness & TroubleshootingNetwork State Awareness & Troubleshooting
Network State Awareness & Troubleshooting
APNIC
 
Link labs LTE-M NB-IOT Hype Webinar slides
Link labs LTE-M NB-IOT Hype Webinar slidesLink labs LTE-M NB-IOT Hype Webinar slides
Link labs LTE-M NB-IOT Hype Webinar slides
Brian Ray
 
Data Center Network Trends - Lin Nease
Data Center Network Trends - Lin NeaseData Center Network Trends - Lin Nease
Data Center Network Trends - Lin Nease
HPDutchWorld
 
IP Signal Distribution
IP Signal DistributionIP Signal Distribution
IP Signal Distribution
rAVe [PUBS]
 
ICS Performance Lab
ICS Performance LabICS Performance Lab
ICS Performance Lab
Jim Gilsinn
 
Accelerated development in Automotive E/E Systems using VisualSim Architect
Accelerated development in Automotive E/E Systems using VisualSim ArchitectAccelerated development in Automotive E/E Systems using VisualSim Architect
Accelerated development in Automotive E/E Systems using VisualSim Architect
Deepak Shankar
 
Mastering IoT Design: Sense, Process, Connect: Processing: Turning IoT Data i...
Mastering IoT Design: Sense, Process, Connect: Processing: Turning IoT Data i...Mastering IoT Design: Sense, Process, Connect: Processing: Turning IoT Data i...
Mastering IoT Design: Sense, Process, Connect: Processing: Turning IoT Data i...
Deepak Shankar
 
Open Programmable Architecture for Java-enabled Network Devices
Open Programmable Architecture for Java-enabled Network DevicesOpen Programmable Architecture for Java-enabled Network Devices
Open Programmable Architecture for Java-enabled Network Devices
Tal Lavian Ph.D.
 
Open programmable architecture for java enabled network devices
Open programmable architecture for java enabled network devicesOpen programmable architecture for java enabled network devices
Open programmable architecture for java enabled network devices
Tal Lavian Ph.D.
 
ADG EOIR Focal Plane Array Sensor Systems Test Overview (1).pptx
ADG EOIR Focal Plane Array Sensor Systems Test Overview (1).pptxADG EOIR Focal Plane Array Sensor Systems Test Overview (1).pptx
ADG EOIR Focal Plane Array Sensor Systems Test Overview (1).pptx
Jatan Shah
 
Considerations when implementing_ha_in_dmf
Considerations when implementing_ha_in_dmfConsiderations when implementing_ha_in_dmf
Considerations when implementing_ha_in_dmf
hik_lhz
 
New Generation Oracle RAC Performance
New Generation Oracle RAC PerformanceNew Generation Oracle RAC Performance
New Generation Oracle RAC Performance
Anil Nair
 
Packet Analysis - Course Technology Computing Conference
Packet Analysis - Course Technology Computing ConferencePacket Analysis - Course Technology Computing Conference
Packet Analysis - Course Technology Computing Conference
Cengage Learning
 
Industrial Ethernet, Part 2: Case Studies
Industrial Ethernet,Part 2: Case StudiesIndustrial Ethernet,Part 2: Case Studies
Industrial Ethernet, Part 2: Case Studies
ControlEng
 
6 profiling tools
6 profiling tools6 profiling tools
6 profiling tools
videos
 
Realtime traffic analyser
Realtime traffic analyserRealtime traffic analyser
Realtime traffic analyser
Alex Moskvin
 
Tech 2 Tech: Network performance
Tech 2 Tech: Network performanceTech 2 Tech: Network performance
Tech 2 Tech: Network performance
Jisc
 
Network State Awareness & Troubleshooting
Network State Awareness & TroubleshootingNetwork State Awareness & Troubleshooting
Network State Awareness & Troubleshooting
APNIC
 
Link labs LTE-M NB-IOT Hype Webinar slides
Link labs LTE-M NB-IOT Hype Webinar slidesLink labs LTE-M NB-IOT Hype Webinar slides
Link labs LTE-M NB-IOT Hype Webinar slides
Brian Ray
 
Data Center Network Trends - Lin Nease
Data Center Network Trends - Lin NeaseData Center Network Trends - Lin Nease
Data Center Network Trends - Lin Nease
HPDutchWorld
 
IP Signal Distribution
IP Signal DistributionIP Signal Distribution
IP Signal Distribution
rAVe [PUBS]
 
ICS Performance Lab
ICS Performance LabICS Performance Lab
ICS Performance Lab
Jim Gilsinn
 
Accelerated development in Automotive E/E Systems using VisualSim Architect
Accelerated development in Automotive E/E Systems using VisualSim ArchitectAccelerated development in Automotive E/E Systems using VisualSim Architect
Accelerated development in Automotive E/E Systems using VisualSim Architect
Deepak Shankar
 
Mastering IoT Design: Sense, Process, Connect: Processing: Turning IoT Data i...
Mastering IoT Design: Sense, Process, Connect: Processing: Turning IoT Data i...Mastering IoT Design: Sense, Process, Connect: Processing: Turning IoT Data i...
Mastering IoT Design: Sense, Process, Connect: Processing: Turning IoT Data i...
Deepak Shankar
 
Open Programmable Architecture for Java-enabled Network Devices
Open Programmable Architecture for Java-enabled Network DevicesOpen Programmable Architecture for Java-enabled Network Devices
Open Programmable Architecture for Java-enabled Network Devices
Tal Lavian Ph.D.
 
Open programmable architecture for java enabled network devices
Open programmable architecture for java enabled network devicesOpen programmable architecture for java enabled network devices
Open programmable architecture for java enabled network devices
Tal Lavian Ph.D.
 
ADG EOIR Focal Plane Array Sensor Systems Test Overview (1).pptx
ADG EOIR Focal Plane Array Sensor Systems Test Overview (1).pptxADG EOIR Focal Plane Array Sensor Systems Test Overview (1).pptx
ADG EOIR Focal Plane Array Sensor Systems Test Overview (1).pptx
Jatan Shah
 
Considerations when implementing_ha_in_dmf
Considerations when implementing_ha_in_dmfConsiderations when implementing_ha_in_dmf
Considerations when implementing_ha_in_dmf
hik_lhz
 
Ad

More from GlobalLogic Ukraine (20)

GlobalLogic JavaScript Community Webinar #21 “Інтерв’ю без заспокійливих”
GlobalLogic JavaScript Community Webinar #21 “Інтерв’ю без заспокійливих”GlobalLogic JavaScript Community Webinar #21 “Інтерв’ю без заспокійливих”
GlobalLogic JavaScript Community Webinar #21 “Інтерв’ю без заспокійливих”
GlobalLogic Ukraine
 
Deadlocks in SQL - Turning Fear Into Understanding (by Sergii Stets)
Deadlocks in SQL - Turning Fear Into Understanding (by Sergii Stets)Deadlocks in SQL - Turning Fear Into Understanding (by Sergii Stets)
Deadlocks in SQL - Turning Fear Into Understanding (by Sergii Stets)
GlobalLogic Ukraine
 
GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...
GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...
GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...
GlobalLogic Ukraine
 
GlobalLogic Embedded Community x ROS Ukraine Webinar "Surgical Robots"
GlobalLogic Embedded Community x ROS Ukraine Webinar "Surgical Robots"GlobalLogic Embedded Community x ROS Ukraine Webinar "Surgical Robots"
GlobalLogic Embedded Community x ROS Ukraine Webinar "Surgical Robots"
GlobalLogic Ukraine
 
GlobalLogic Java Community Webinar #17 “SpringJDBC vs JDBC. Is Spring a Hero?”
GlobalLogic Java Community Webinar #17 “SpringJDBC vs JDBC. Is Spring a Hero?”GlobalLogic Java Community Webinar #17 “SpringJDBC vs JDBC. Is Spring a Hero?”
GlobalLogic Java Community Webinar #17 “SpringJDBC vs JDBC. Is Spring a Hero?”
GlobalLogic Ukraine
 
GlobalLogic JavaScript Community Webinar #18 “Long Story Short: OSI Model”
GlobalLogic JavaScript Community Webinar #18 “Long Story Short: OSI Model”GlobalLogic JavaScript Community Webinar #18 “Long Story Short: OSI Model”
GlobalLogic JavaScript Community Webinar #18 “Long Story Short: OSI Model”
GlobalLogic Ukraine
 
Штучний інтелект як допомога в навчанні, а не замінник.pptx
Штучний інтелект як допомога в навчанні, а не замінник.pptxШтучний інтелект як допомога в навчанні, а не замінник.pptx
Штучний інтелект як допомога в навчанні, а не замінник.pptx
GlobalLogic Ukraine
 
Задачі AI-розробника як застосовується штучний інтелект.pptx
Задачі AI-розробника як застосовується штучний інтелект.pptxЗадачі AI-розробника як застосовується штучний інтелект.pptx
Задачі AI-розробника як застосовується штучний інтелект.pptx
GlobalLogic Ukraine
 
Що треба вивчати, щоб стати розробником штучного інтелекту та нейромереж.pptx
Що треба вивчати, щоб стати розробником штучного інтелекту та нейромереж.pptxЩо треба вивчати, щоб стати розробником штучного інтелекту та нейромереж.pptx
Що треба вивчати, щоб стати розробником штучного інтелекту та нейромереж.pptx
GlobalLogic Ukraine
 
GlobalLogic Java Community Webinar #16 “Zaloni’s Architecture for Data-Driven...
GlobalLogic Java Community Webinar #16 “Zaloni’s Architecture for Data-Driven...GlobalLogic Java Community Webinar #16 “Zaloni’s Architecture for Data-Driven...
GlobalLogic Java Community Webinar #16 “Zaloni’s Architecture for Data-Driven...
GlobalLogic Ukraine
 
JavaScript Community Webinar #14 "Why Is Git Rebase?"
JavaScript Community Webinar #14 "Why Is Git Rebase?"JavaScript Community Webinar #14 "Why Is Git Rebase?"
JavaScript Community Webinar #14 "Why Is Git Rebase?"
GlobalLogic Ukraine
 
GlobalLogic .NET Community Webinar #3 "Exploring Serverless with Azure Functi...
GlobalLogic .NET Community Webinar #3 "Exploring Serverless with Azure Functi...GlobalLogic .NET Community Webinar #3 "Exploring Serverless with Azure Functi...
GlobalLogic .NET Community Webinar #3 "Exploring Serverless with Azure Functi...
GlobalLogic Ukraine
 
Страх і сила помилок - IT Inside від GlobalLogic Education
Страх і сила помилок - IT Inside від GlobalLogic EducationСтрах і сила помилок - IT Inside від GlobalLogic Education
Страх і сила помилок - IT Inside від GlobalLogic Education
GlobalLogic Ukraine
 
GlobalLogic .NET Webinar #2 “Azure RBAC and Managed Identity”
GlobalLogic .NET Webinar #2 “Azure RBAC and Managed Identity”GlobalLogic .NET Webinar #2 “Azure RBAC and Managed Identity”
GlobalLogic .NET Webinar #2 “Azure RBAC and Managed Identity”
GlobalLogic Ukraine
 
GlobalLogic QA Webinar “What does it take to become a Test Engineer”
GlobalLogic QA Webinar “What does it take to become a Test Engineer”GlobalLogic QA Webinar “What does it take to become a Test Engineer”
GlobalLogic QA Webinar “What does it take to become a Test Engineer”
GlobalLogic Ukraine
 
“How to Secure Your Applications With a Keycloak?
“How to Secure Your Applications With a Keycloak?“How to Secure Your Applications With a Keycloak?
“How to Secure Your Applications With a Keycloak?
GlobalLogic Ukraine
 
GlobalLogic Machine Learning Webinar “Advanced Statistical Methods for Linear...
GlobalLogic Machine Learning Webinar “Advanced Statistical Methods for Linear...GlobalLogic Machine Learning Webinar “Advanced Statistical Methods for Linear...
GlobalLogic Machine Learning Webinar “Advanced Statistical Methods for Linear...
GlobalLogic Ukraine
 
GlobalLogic Machine Learning Webinar “Statistical learning of linear regressi...
GlobalLogic Machine Learning Webinar “Statistical learning of linear regressi...GlobalLogic Machine Learning Webinar “Statistical learning of linear regressi...
GlobalLogic Machine Learning Webinar “Statistical learning of linear regressi...
GlobalLogic Ukraine
 
GlobalLogic C++ Webinar “The Minimum Knowledge to Become a C++ Developer”
GlobalLogic C++ Webinar “The Minimum Knowledge to Become a C++ Developer”GlobalLogic C++ Webinar “The Minimum Knowledge to Become a C++ Developer”
GlobalLogic C++ Webinar “The Minimum Knowledge to Become a C++ Developer”
GlobalLogic Ukraine
 
GlobalLogic Webinar "Introduction to Embedded QA"
GlobalLogic Webinar "Introduction to Embedded QA"GlobalLogic Webinar "Introduction to Embedded QA"
GlobalLogic Webinar "Introduction to Embedded QA"
GlobalLogic Ukraine
 
GlobalLogic JavaScript Community Webinar #21 “Інтерв’ю без заспокійливих”
GlobalLogic JavaScript Community Webinar #21 “Інтерв’ю без заспокійливих”GlobalLogic JavaScript Community Webinar #21 “Інтерв’ю без заспокійливих”
GlobalLogic JavaScript Community Webinar #21 “Інтерв’ю без заспокійливих”
GlobalLogic Ukraine
 
Deadlocks in SQL - Turning Fear Into Understanding (by Sergii Stets)
Deadlocks in SQL - Turning Fear Into Understanding (by Sergii Stets)Deadlocks in SQL - Turning Fear Into Understanding (by Sergii Stets)
Deadlocks in SQL - Turning Fear Into Understanding (by Sergii Stets)
GlobalLogic Ukraine
 
GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...
GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...
GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...
GlobalLogic Ukraine
 
GlobalLogic Embedded Community x ROS Ukraine Webinar "Surgical Robots"
GlobalLogic Embedded Community x ROS Ukraine Webinar "Surgical Robots"GlobalLogic Embedded Community x ROS Ukraine Webinar "Surgical Robots"
GlobalLogic Embedded Community x ROS Ukraine Webinar "Surgical Robots"
GlobalLogic Ukraine
 
GlobalLogic Java Community Webinar #17 “SpringJDBC vs JDBC. Is Spring a Hero?”
GlobalLogic Java Community Webinar #17 “SpringJDBC vs JDBC. Is Spring a Hero?”GlobalLogic Java Community Webinar #17 “SpringJDBC vs JDBC. Is Spring a Hero?”
GlobalLogic Java Community Webinar #17 “SpringJDBC vs JDBC. Is Spring a Hero?”
GlobalLogic Ukraine
 
GlobalLogic JavaScript Community Webinar #18 “Long Story Short: OSI Model”
GlobalLogic JavaScript Community Webinar #18 “Long Story Short: OSI Model”GlobalLogic JavaScript Community Webinar #18 “Long Story Short: OSI Model”
GlobalLogic JavaScript Community Webinar #18 “Long Story Short: OSI Model”
GlobalLogic Ukraine
 
Штучний інтелект як допомога в навчанні, а не замінник.pptx
Штучний інтелект як допомога в навчанні, а не замінник.pptxШтучний інтелект як допомога в навчанні, а не замінник.pptx
Штучний інтелект як допомога в навчанні, а не замінник.pptx
GlobalLogic Ukraine
 
Задачі AI-розробника як застосовується штучний інтелект.pptx
Задачі AI-розробника як застосовується штучний інтелект.pptxЗадачі AI-розробника як застосовується штучний інтелект.pptx
Задачі AI-розробника як застосовується штучний інтелект.pptx
GlobalLogic Ukraine
 
Що треба вивчати, щоб стати розробником штучного інтелекту та нейромереж.pptx
Що треба вивчати, щоб стати розробником штучного інтелекту та нейромереж.pptxЩо треба вивчати, щоб стати розробником штучного інтелекту та нейромереж.pptx
Що треба вивчати, щоб стати розробником штучного інтелекту та нейромереж.pptx
GlobalLogic Ukraine
 
GlobalLogic Java Community Webinar #16 “Zaloni’s Architecture for Data-Driven...
GlobalLogic Java Community Webinar #16 “Zaloni’s Architecture for Data-Driven...GlobalLogic Java Community Webinar #16 “Zaloni’s Architecture for Data-Driven...
GlobalLogic Java Community Webinar #16 “Zaloni’s Architecture for Data-Driven...
GlobalLogic Ukraine
 
JavaScript Community Webinar #14 "Why Is Git Rebase?"
JavaScript Community Webinar #14 "Why Is Git Rebase?"JavaScript Community Webinar #14 "Why Is Git Rebase?"
JavaScript Community Webinar #14 "Why Is Git Rebase?"
GlobalLogic Ukraine
 
GlobalLogic .NET Community Webinar #3 "Exploring Serverless with Azure Functi...
GlobalLogic .NET Community Webinar #3 "Exploring Serverless with Azure Functi...GlobalLogic .NET Community Webinar #3 "Exploring Serverless with Azure Functi...
GlobalLogic .NET Community Webinar #3 "Exploring Serverless with Azure Functi...
GlobalLogic Ukraine
 
Страх і сила помилок - IT Inside від GlobalLogic Education
Страх і сила помилок - IT Inside від GlobalLogic EducationСтрах і сила помилок - IT Inside від GlobalLogic Education
Страх і сила помилок - IT Inside від GlobalLogic Education
GlobalLogic Ukraine
 
GlobalLogic .NET Webinar #2 “Azure RBAC and Managed Identity”
GlobalLogic .NET Webinar #2 “Azure RBAC and Managed Identity”GlobalLogic .NET Webinar #2 “Azure RBAC and Managed Identity”
GlobalLogic .NET Webinar #2 “Azure RBAC and Managed Identity”
GlobalLogic Ukraine
 
GlobalLogic QA Webinar “What does it take to become a Test Engineer”
GlobalLogic QA Webinar “What does it take to become a Test Engineer”GlobalLogic QA Webinar “What does it take to become a Test Engineer”
GlobalLogic QA Webinar “What does it take to become a Test Engineer”
GlobalLogic Ukraine
 
“How to Secure Your Applications With a Keycloak?
“How to Secure Your Applications With a Keycloak?“How to Secure Your Applications With a Keycloak?
“How to Secure Your Applications With a Keycloak?
GlobalLogic Ukraine
 
GlobalLogic Machine Learning Webinar “Advanced Statistical Methods for Linear...
GlobalLogic Machine Learning Webinar “Advanced Statistical Methods for Linear...GlobalLogic Machine Learning Webinar “Advanced Statistical Methods for Linear...
GlobalLogic Machine Learning Webinar “Advanced Statistical Methods for Linear...
GlobalLogic Ukraine
 
GlobalLogic Machine Learning Webinar “Statistical learning of linear regressi...
GlobalLogic Machine Learning Webinar “Statistical learning of linear regressi...GlobalLogic Machine Learning Webinar “Statistical learning of linear regressi...
GlobalLogic Machine Learning Webinar “Statistical learning of linear regressi...
GlobalLogic Ukraine
 
GlobalLogic C++ Webinar “The Minimum Knowledge to Become a C++ Developer”
GlobalLogic C++ Webinar “The Minimum Knowledge to Become a C++ Developer”GlobalLogic C++ Webinar “The Minimum Knowledge to Become a C++ Developer”
GlobalLogic C++ Webinar “The Minimum Knowledge to Become a C++ Developer”
GlobalLogic Ukraine
 
GlobalLogic Webinar "Introduction to Embedded QA"
GlobalLogic Webinar "Introduction to Embedded QA"GlobalLogic Webinar "Introduction to Embedded QA"
GlobalLogic Webinar "Introduction to Embedded QA"
GlobalLogic Ukraine
 
Ad

Recently uploaded (20)

Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Everything You Need to Know About Agentforce? (Put AI Agents to Work)Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Cyntexa
 
Crazy Incentives and How They Kill Security. How Do You Turn the Wheel?
Crazy Incentives and How They Kill Security. How Do You Turn the Wheel?Crazy Incentives and How They Kill Security. How Do You Turn the Wheel?
Crazy Incentives and How They Kill Security. How Do You Turn the Wheel?
Christian Folini
 
Kit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdf
Kit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdfKit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdf
Kit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdf
Wonjun Hwang
 
Kit-Works Team Study_아직도 Dockefile.pdf_김성호
Kit-Works Team Study_아직도 Dockefile.pdf_김성호Kit-Works Team Study_아직도 Dockefile.pdf_김성호
Kit-Works Team Study_아직도 Dockefile.pdf_김성호
Wonjun Hwang
 
Integrating FME with Python: Tips, Demos, and Best Practices for Powerful Aut...
Integrating FME with Python: Tips, Demos, and Best Practices for Powerful Aut...Integrating FME with Python: Tips, Demos, and Best Practices for Powerful Aut...
Integrating FME with Python: Tips, Demos, and Best Practices for Powerful Aut...
Safe Software
 
Unlocking Generative AI in your Web Apps
Unlocking Generative AI in your Web AppsUnlocking Generative AI in your Web Apps
Unlocking Generative AI in your Web Apps
Maximiliano Firtman
 
Bepents tech services - a premier cybersecurity consulting firm
Bepents tech services - a premier cybersecurity consulting firmBepents tech services - a premier cybersecurity consulting firm
Bepents tech services - a premier cybersecurity consulting firm
Benard76
 
How to Install & Activate ListGrabber - eGrabber
How to Install & Activate ListGrabber - eGrabberHow to Install & Activate ListGrabber - eGrabber
How to Install & Activate ListGrabber - eGrabber
eGrabber
 
AI 3-in-1: Agents, RAG, and Local Models - Brent Laster
AI 3-in-1: Agents, RAG, and Local Models - Brent LasterAI 3-in-1: Agents, RAG, and Local Models - Brent Laster
AI 3-in-1: Agents, RAG, and Local Models - Brent Laster
All Things Open
 
Dark Dynamism: drones, dark factories and deurbanization
Dark Dynamism: drones, dark factories and deurbanizationDark Dynamism: drones, dark factories and deurbanization
Dark Dynamism: drones, dark factories and deurbanization
Jakub Šimek
 
Developing System Infrastructure Design Plan.pptx
Developing System Infrastructure Design Plan.pptxDeveloping System Infrastructure Design Plan.pptx
Developing System Infrastructure Design Plan.pptx
wondimagegndesta
 
Viam product demo_ Deploying and scaling AI with hardware.pdf
Viam product demo_ Deploying and scaling AI with hardware.pdfViam product demo_ Deploying and scaling AI with hardware.pdf
Viam product demo_ Deploying and scaling AI with hardware.pdf
camilalamoratta
 
Top-AI-Based-Tools-for-Game-Developers (1).pptx
Top-AI-Based-Tools-for-Game-Developers (1).pptxTop-AI-Based-Tools-for-Game-Developers (1).pptx
Top-AI-Based-Tools-for-Game-Developers (1).pptx
BR Softech
 
Config 2025 presentation recap covering both days
Config 2025 presentation recap covering both daysConfig 2025 presentation recap covering both days
Config 2025 presentation recap covering both days
TrishAntoni1
 
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Mike Mingos
 
machines-for-woodworking-shops-en-compressed.pdf
machines-for-woodworking-shops-en-compressed.pdfmachines-for-woodworking-shops-en-compressed.pdf
machines-for-woodworking-shops-en-compressed.pdf
AmirStern2
 
Mastering Testing in the Modern F&B Landscape
Mastering Testing in the Modern F&B LandscapeMastering Testing in the Modern F&B Landscape
Mastering Testing in the Modern F&B Landscape
marketing943205
 
AI-proof your career by Olivier Vroom and David WIlliamson
AI-proof your career by Olivier Vroom and David WIlliamsonAI-proof your career by Olivier Vroom and David WIlliamson
AI-proof your career by Olivier Vroom and David WIlliamson
UXPA Boston
 
Building the Customer Identity Community, Together.pdf
Building the Customer Identity Community, Together.pdfBuilding the Customer Identity Community, Together.pdf
Building the Customer Identity Community, Together.pdf
Cheryl Hung
 
Build With AI - In Person Session Slides.pdf
Build With AI - In Person Session Slides.pdfBuild With AI - In Person Session Slides.pdf
Build With AI - In Person Session Slides.pdf
Google Developer Group - Harare
 
Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Everything You Need to Know About Agentforce? (Put AI Agents to Work)Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Cyntexa
 
Crazy Incentives and How They Kill Security. How Do You Turn the Wheel?
Crazy Incentives and How They Kill Security. How Do You Turn the Wheel?Crazy Incentives and How They Kill Security. How Do You Turn the Wheel?
Crazy Incentives and How They Kill Security. How Do You Turn the Wheel?
Christian Folini
 
Kit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdf
Kit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdfKit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdf
Kit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdf
Wonjun Hwang
 
Kit-Works Team Study_아직도 Dockefile.pdf_김성호
Kit-Works Team Study_아직도 Dockefile.pdf_김성호Kit-Works Team Study_아직도 Dockefile.pdf_김성호
Kit-Works Team Study_아직도 Dockefile.pdf_김성호
Wonjun Hwang
 
Integrating FME with Python: Tips, Demos, and Best Practices for Powerful Aut...
Integrating FME with Python: Tips, Demos, and Best Practices for Powerful Aut...Integrating FME with Python: Tips, Demos, and Best Practices for Powerful Aut...
Integrating FME with Python: Tips, Demos, and Best Practices for Powerful Aut...
Safe Software
 
Unlocking Generative AI in your Web Apps
Unlocking Generative AI in your Web AppsUnlocking Generative AI in your Web Apps
Unlocking Generative AI in your Web Apps
Maximiliano Firtman
 
Bepents tech services - a premier cybersecurity consulting firm
Bepents tech services - a premier cybersecurity consulting firmBepents tech services - a premier cybersecurity consulting firm
Bepents tech services - a premier cybersecurity consulting firm
Benard76
 
How to Install & Activate ListGrabber - eGrabber
How to Install & Activate ListGrabber - eGrabberHow to Install & Activate ListGrabber - eGrabber
How to Install & Activate ListGrabber - eGrabber
eGrabber
 
AI 3-in-1: Agents, RAG, and Local Models - Brent Laster
AI 3-in-1: Agents, RAG, and Local Models - Brent LasterAI 3-in-1: Agents, RAG, and Local Models - Brent Laster
AI 3-in-1: Agents, RAG, and Local Models - Brent Laster
All Things Open
 
Dark Dynamism: drones, dark factories and deurbanization
Dark Dynamism: drones, dark factories and deurbanizationDark Dynamism: drones, dark factories and deurbanization
Dark Dynamism: drones, dark factories and deurbanization
Jakub Šimek
 
Developing System Infrastructure Design Plan.pptx
Developing System Infrastructure Design Plan.pptxDeveloping System Infrastructure Design Plan.pptx
Developing System Infrastructure Design Plan.pptx
wondimagegndesta
 
Viam product demo_ Deploying and scaling AI with hardware.pdf
Viam product demo_ Deploying and scaling AI with hardware.pdfViam product demo_ Deploying and scaling AI with hardware.pdf
Viam product demo_ Deploying and scaling AI with hardware.pdf
camilalamoratta
 
Top-AI-Based-Tools-for-Game-Developers (1).pptx
Top-AI-Based-Tools-for-Game-Developers (1).pptxTop-AI-Based-Tools-for-Game-Developers (1).pptx
Top-AI-Based-Tools-for-Game-Developers (1).pptx
BR Softech
 
Config 2025 presentation recap covering both days
Config 2025 presentation recap covering both daysConfig 2025 presentation recap covering both days
Config 2025 presentation recap covering both days
TrishAntoni1
 
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Mike Mingos
 
machines-for-woodworking-shops-en-compressed.pdf
machines-for-woodworking-shops-en-compressed.pdfmachines-for-woodworking-shops-en-compressed.pdf
machines-for-woodworking-shops-en-compressed.pdf
AmirStern2
 
Mastering Testing in the Modern F&B Landscape
Mastering Testing in the Modern F&B LandscapeMastering Testing in the Modern F&B Landscape
Mastering Testing in the Modern F&B Landscape
marketing943205
 
AI-proof your career by Olivier Vroom and David WIlliamson
AI-proof your career by Olivier Vroom and David WIlliamsonAI-proof your career by Olivier Vroom and David WIlliamson
AI-proof your career by Olivier Vroom and David WIlliamson
UXPA Boston
 
Building the Customer Identity Community, Together.pdf
Building the Customer Identity Community, Together.pdfBuilding the Customer Identity Community, Together.pdf
Building the Customer Identity Community, Together.pdf
Cheryl Hung
 

Embedded Webinar #17 "Low-level Network Testing in Embedded Devices Development"

  • 1. Network Security in Embedded devices Serhii Korniienko - Embedded QA engineer
  • 2. Examples of properties of typical embedded computers when compared with general-purpose counterparts are: • low power consumption • small size • rugged operating ranges • low per-unit cost • often ‘non-standard‘ solution
  • 3. Why low-level network testing is a must in embedded: • Often custom protocols used to reduce CPU/RAM/ROM/bandwidth usage • Low cost • Timing properties of the program depend heavily on its environment • Environment = +Process or & Memory Hierarchy + Operating System, other processes/threads, … + Network + I/O Devices +…
  • 4. Our experience: 1. Requirements 2. Approach 3. Tools 4. Test setup 5. Logs gathering and analysis 6. Issues 7. Test cases 8. Process
  • 5. 1. Our requirements: • Low cost – cheapest HW components approved by customer (BoM < 20$) • Http and custom UDP support • Stack: freeRTOS + LwIP • Rigid requirements on synchronization jitter (PTP) • IC network switch to build daisy-chained network structure • Stable work in heavy loaded network (1Gb wireshark dump file per minute)
  • 6. 2. Low –level testing approach: • List of all low-level network stuff: – Ports inventory: – 80/tcp – http – 554/tcp - rtsp – 67/udp - dhcps – 68/udp - dhcpc – 319/udp - ptp-event – 320/udp - ptp-general – 2467/udp - custom protocol – 17100/udp - custom protocol
  • 8. 3. Tools: 1. NMAP – enumeration and packet sending - sS - sU - sO - sX - sN - sF - sX - sA - sY - --scanflags
  • 9. 3. Tools: 2. Ostinato –packet editing and traffic generation - Set a value for any field of any protocol - Open and edit PCAP files, replay and save back - Support for the most common standard protocols - Free, Libre, Open-Source Software
  • 10. 3. Tools: 3. LOIC – open-source network stress testing and denial-of-service attack application - IC Switch throughput test
  • 12. 5. Logs gathering and analysis: The key thing is to gather and analyze all available info from all system components (switch and microcontroller) to recognize and eliminate bottlenecks and issues: - Buffer overflows and underflows - Queue overflows and underflows - Protocol/link state changes - Tx/Rx error counters - Switch counters - Error messages - Power dropdown - Unauthorized recording - Voltage levels - …
  • 13. 6. Issues found: Issue Web server goes down during polite TCP scan Discovered/ Investigated NMAP scan Root cause LwIP configuration issue
  • 14. Issues found: Issue Discovered/ investigated Root cause Hard-fault occurs if send fragmented UDP-packet NMAP/ Wireshark/ Ostinato LwIP incorrectly reassemble fragmented UDP packets
  • 15. Issues found: Issue Discovered/ investigated Root cause Device can be shutdown by specially crafted control message Script/fuzzing Input validation issue Device can turn off it’s power supply and network chip power supply ☺
  • 16. Issues found: Issue Device hard-fault if send abnormally large or small message on control port - Discovered/ investigated LOIC/ Script/fuzzing Root cause Input validation issue
  • 17. Issues found: Issue Discovered/ investigated Root cause Sending short ptp packet cause ptp recalibration and packet loss NMAP scan Ostinato LwIP issue
  • 18. Issues found: Issue Discovered/ investigated Root cause Network link goes down and Up during nmap scan NMAP/ Wireshark/ Ostinato PSU issue for new IC switch Network link goes down during nmap scan NMAP/ Wireshark/ Ostinato unauthorized command on switch management interface
  • 19. Issues found: Issue Discovered/ investigated Root cause Device become unreachable if send short UDP packet on port 319/320 NMAP scan Ostinato IC switch issue All devices in chain become unavailable if perform UDP flood on port 319/320 of last device in chain LOIC/Ostinato IC switch issue
  • 23. 8. Process: • Perform low-level testing to every hardware changes • Preform low-level testing for every configuration changes • Preform low-level testing for every low-level structures changes • Preform low-level testing periodically
  • 24. Conclusion: If you still hesitate about the necessity to perform low-level network testing – go to the issues found chapter!
  翻译: