YY/T 1833.3-2022 PDF EnglishUS$500.00 · In stock · Download in 9 seconds
YY/T 1833.3-2022: Artificial intelligence medical device - Quality requirements and evaluation - Part 3: General requirement for data annotation Delivery: 9 seconds. True-PDF full-copy in English & invoice will be downloaded + auto-delivered via email. See step-by-step procedure Status: Valid
Similar standardsYY/T 1833.3-2022: Artificial intelligence medical device - Quality requirements and evaluation - Part 3: General requirement for data annotation---This is an excerpt. Full copy of true-PDF in English version (including equations, symbols, images, flow-chart, tables, and figures etc.), auto-downloaded/delivered in 9 seconds, can be purchased online: https://www.ChineseStandard.net/PDF.aspx/YYT1833.3-2022 YY PHARMACEUTICAL INDUSTRY STANDARD ICS 11.040.99 CCS C 40 Artificial intelligence medical device - Quality requirements and evaluation -- Part 3.General requirement for data annotation Issued on: AUGUST 17, 2022 Implemented on: SEPTEMBER 01, 2023 Issued by. National Medical Products Administration Table of ContentsForeword... 3 Introduction... 4 1 Scope... 5 2 Normative references... 5 3 Terms and definitions... 5 4 Documentation for annotation tasks... 7 5 Quality characteristics of data annotation... 9 6 Annotation and quality control process... 11 7 Annotation tools... 12 8 Evaluation method... 16 Annex A (informative) Description examples for annotation task... 18 Annex B (informative) Business architecture examples (chest CT lung nodules)... 34 Annex C (informative) Evaluation of AI-assisted annotation performance... 37 Bibliography... 47 Artificial intelligence medical device - Quality requirements and evaluation - Part 3.General requirement for data annotation1 ScopeThis document specifies the general requirements and evaluation methods for data labeling of artificial intelligence medical devices. This document applies to data labeling activities for artificial intelligence medical devices.2 Normative referencesThe following referenced documents are indispensable for the application of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies. YY/T 1833.1, Artificial intelligence medical device - Quality requirements and evaluation - Part 1.Terminology YY/T 1833.2, Artificial intelligence medical device - Quality requirements and evaluation - Part 2.General requirements for datasets3 Terms and definitionsFor the purposes of this document, the terms and definitions defined in YY/T 1833.1, YY/T 1833.2 as well as the followings apply. 3.1 annotation task The activity of purposefully analyzing a batch of data and adding external knowledge. 3.2 annotation object The detailed information of annotation task analysis, such as the type, characteristics, and attributes of the data. 3.3 structured annotation annotators. 3.14 annotator performance A representation of the ability of annotators to perform annotation tasks. 3.15 annotation responsible organization The entity that organizes and carries out annotation tasks and has direct responsibility for the quality of annotation.4 Documentation for annotation tasks4.1 Classification of annotation tasks Before the annotation task begins, the annotation responsible organization shall clearly define the classification of the annotation task, including dimensions such as data modality, execution subject, annotation result format, annotation result nature, and annotation result form. The data modalities of annotation tasks are divided into images, signals, videos, texts, etc. According to the execution subject, annotation tasks can be divided into manual annotation, automatic annotation, semi-automatic annotation, etc. According to the format of the annotation results, annotation tasks can be divided into structured annotation, non-structured annotation, semi-structured annotation, etc. The nature of the annotation results can be divided into GT value, reference standard, gold standard, etc. The form of the annotation results can be divided into detection, classification, segmentation, semantics, etc. NOTE. Semantic annotation is often used to describe the relationship or connection between objects, such as the relative position of muscle and fat on ultrasound images. 4.2 Description of annotation task 4.2.1 Annotation rules The annotation responsible organization shall state the rules on which the annotation task is based, meeting the following requirements. - The definition of each annotation object is unique and unambiguous; - The name of the annotation object has a compliance document; - Different annotation objects are distinguishable; - The qualitative characteristics of the annotation object shall be verifiable; - Definition and examples of annotation objects, such as positive samples, negative samples, target areas, non-target areas, major signs, minor signs, interference items, examples of difficult situations, etc.; - Storage format, preview method, granularity, accuracy, etc. of annotation results and measurement results. The annotation responsible organization shall describe the data organization plan, such as data cleaning, data duplication checking, etc. For data from laboratory measurements, the annotation responsible organization shall describe the measurement method, measurement device, measurement conditions and personnel, etc. For data from simulation synthesis, the annotation responsible organization shall describe the calculation process and confirmation method. NOTE. Annex A gives an example of a annotation task description document.5 Quality characteristics of data annotation5.1 Accuracy The annotation responsible organization shall declare the accuracy of the annotation results based on the form of the annotation results. If applicable, in specific annotation scenarios, the following indicators can be used. - Detection. recall rate, precision; - Classification. sensitivity, specificity, accuracy; - Segmentation. Dice coefficient, intersection-over-union ratio, Hausdorff distance; - Measurement, counting. absolute error, relative error; - Dynamic curve evaluation. Pearson correlation coefficient, 2-norm error. 5.2 Consistency The annotation responsible organization shall declare the internal consistency of input and output data and information in each link of the annotation process, including personnel information, annotation results, and original data. The annotation responsible organization shall declare the consistency between annotators, such as. Classification task. Use Kappa coefficient to describe the consistency between annotators drops significantly, the annotation responsible organization shall rest, train and re-evaluate the annotators. Repeatability can be evaluated by using the method of question verification, which counts the annotation results of the same annotator on the same data in each continuous annotation process. The proportion of samples with consistent repeated annotation or errors within the allowable range in the repeated annotation samples is calculated. NOTE 1.For example, after completing the classification and annotation of 20 diabetic retinopathy images, one of them is randomly selected and re-annotated. The accuracy index can be evaluated by comparing the annotators' and the arbitration conclusions. The proportion of primary annotation samples that the arbitration personnel consider correct is calculated. NOTE 2.For example, after every 20 diabetic retinopathy images are annotated, one image is randomly selected for arbitration and comparison by an arbitrator to ensure statistical accuracy. 6.2.5 Security management The annotation responsible organization shall implement the following safety management measures. a) Before annotation, the annotation responsible organization shall ensure that the data to be annotated has been desensitized. An independent backup of the data to be annotated shall be established to ensure that the backup is not modified or deleted; b) Before the equipment that performs data annotation, calculation and storage is deactivated, retired or exits the annotation task, all data in it shall be completely deleted and cannot be restored; c) The party responsible for annotation shall ensure the network security of the annotation process, such as using firewalls, boundary protection, intrusion protection and other security measures.7 Annotation tools7.1 Functions 7.1.1 Processing objects The annotation tool shall clearly define the scope of the processing object, including data collection method and storage format. a) According to the data collection method, the processing objects can be divided into. - Image data. CT, MR, PET, X-ray, mammography, ultrasound, endoscopy, pathology, etc.; - Signal data. electrocardiogram (ECG), electroencephalogram (EEG), electromyogram (EMG), heart and lung sounds, etc.; - Text data (if applicable). outpatient and emergency records, hospitalization records, laboratory records, medication records, surgical records, and follow- up records. b) According to the data storage format, the processing objects can be divided into. - Image format. Dicom, Dicom-RT, png, jpg, tif, etc.; - Signal format. xml, HL7, etc.; - Video format. avi, mp4, etc.; - Text format (if applicable). txt, doc, pdf, etc.; - Other formats. data formats customized by the manufacturer. 7.1.2 Data display The annotation tool shall have a data display interface that meets the following requirements. a) The annotation tool shall support data display functions within the data reading range, such as. - Dicom format data. sequence page turning, window width and window position adjustment, multi-pane display, translation, overall zoom, inversion, local zoom, straight line measurement, angle measurement, image rotation/flip, sequence playback, restore original image, image rendering, image gain, dynamic range, etc.; - Video format data. video playback pause, frame rate adjustment, overall zoom, local zoom, contrast adjustment, saturation adjustment, etc.; - Picture format data. translation, rotation, overall zoom, local zoom, contrast adjustment, saturation adjustment, etc.; - Text format data. font size adjustment, font type adjustment, local zoom, single column display, multi-column display, full page display, scrolling display, etc. b) The data display interface shall prevent unauthorized access to data, such as copying, downloading, saving, printing, etc. 7.1.3 Data annotation - Support the setting of result import and export permissions, including personnel permissions, data permissions, project permissions, etc. 7.1.5 Progress display Annotation tools and platforms shall provide the ability to display the progress of annotation tasks, such as. - Support data annotation status display, including unannotated and annotated, etc. - Support project or data set annotation progress statistics and display functions, including percentage display, bar chart display, pie chart display, etc. - Support conditional search annotation progress statistics and display functions, and the search conditions include project, data set, data type, annotator, etc. 7.1.6 Task scheduling The annotation tools and platforms shall have the annotation task scheduling function, such as. - Support the creation, viewing, suspension, resumption, restart, deletion, modification and corresponding permission configuration of annotation tasks; - Support the permission configuration of annotation tasks, including personnel permissions, data permissions, project permissions, operation process permissions, etc.; - Support the logical configuration of annotation tasks, including cross-annotation methods, arbitration annotation conditions and methods, and review annotation conditions and methods. 7.1.7 Review and arbitration For annotation tasks that require review and arbitration, annotation tools and platforms shall support custom configuration functions, such as. - Support custom configuration of arbitration conditions and methods, including arbitration trigger conditions, arbitration personnel settings, and arbitration data settings; - Support custom configuration of audit conditions and methods, including audit trigger conditions, annotation reviewer settings, and audit data settings. 7.1.8 Process records The annotation tools and platforms shall have process recording functions and meet the requirements of 5.8 traceability. 7.1.9 Security features Annotation tools shall have the following safety features. a) Data transmission security. Data transmission shall ensure that data is transmitted to the designated object in a secure manner, such as using encryption technology, identity authentication technology, data integrity verification technology, etc.; b) Data storage security. Annotation tools shall have security measures to ensure data security, such as encrypted storage. Original data and annotation results shall be stored separately as original data files and annotation data files; c) Identity authentication. Users shall be identified and identification information shall be managed and maintained. The uniqueness of users during the life cycle of the information system shall be ensured. Identity authentication shall be successfully performed before the user makes an action request. User login passwords shall be changed regularly; d) Access control. Access control policies shall be in place and control of operations between subjects and objects under policy control shall be implemented.8 Evaluation method8.1 Documentation of annotation tasks Review the documents provided by the annotation responsible organization, which shall meet the requirements of Chapter 4. 8.2 Quality characteristics of annotation tasks 8.2.1 Accuracy In specific annotation scenarios, the annotation results can be sampled according to Chapter 6 of YY/T 1833.2-2022.The sampled samples or all samples are evaluated by expert demonstration, expert comparison, quantitative calculation, etc. The indicators specified by the annotation responsible organization shall be calculated to meet the requirements of 5.1. 8.2.2 Consistency The consistency between the annotation results and the process documents shall be checked by sampling inspection and shall meet the requirements of 5.2. 8.2.3 Precision According to the claim of the party responsible for annotation, check the quantitative characteristics of the data contained in the annotation results, which shall meet theAnnex A(informative) Description examples for annotation task A.1 Wearable ECG A.1.1 Classification of annotation tasks This annotation task belongs to physiological signal annotation according to the data modality. The data modality is a single-lead wearable ECG waveform signal. The execution subject is manual annotation. This annotation task belongs to structured annotation. The storage format of the annotation result is HL7.The annotation result gives the classification of signal quality as a reference standard. A.1.2 Annotation rules The annotation object of this annotation task is the quality of ECG signal (the overall quality of ECG signal in every 10 s). The definition and annotation rules of ECG signal quality are given by an expert group composed of ECG clinical experts and engineering and technical experts based on clinical literature and discussion. The professional titles of experts are all above associate senior. Among them, medical experts have been engaged in clinical work for more than 10 years. They have been engaged in data annotation related work for more than 1 year. The annotation results include two categories, namely "good signal quality" and "poor signal quality". "Good signal quality" is defined as a clear QRS complex in the ECG signal observation window. There is almost no baseline drift, that is, the baseline drift amplitude does not exceed 1/3 of the signal amplitude, and does not affect the QRS wave judgment. The T wave in the observation window is clear, and there are no more than 2 unrecognizable T waves. High-frequency noise interference is minimal. Pathological changes do not affect the judgment of the signal quality level, such as premature beats, tachycardia and other pathological processes. As long as the waveform is clear, it is judged as "good signal quality". ECG signals that do not meet the above conditions are judged as "poor signal quality". The annotation rules are as follows. Organize 3 ECG doctors to train in advance on the definition of ECG signal quality and software operation. During annotation, each doctor uses the software to mark the signal quality back to back. Record the annotation results of each annotator. First, use the minority obeys the majority method, that is, the signal quality result of this segment determined by no less than 2 annotators is used as the initial annotation result of this segment. The annotators review the initial annotation results of the signal quality face to face. If there is no doubt about the initial annotation result, the initial annotation result is used as the final annotation result. If there is a disagreement or doubt about the initial annotation result, the expert group (composed of 3 experts) will be asked to arbitrate. The expert group will give the final annotation result after discussion based on the preliminary annotation results. During the annotation process, a certain segment of the signal can be repeated periodically. Observe whether the results of the annotators are consistent. If there is a contradiction, use rest, training and other means to intervene. A.1.3 Annotators The ECG physician must have been engaged in clinical work for no less than 1 year, have obtained the professional title of physician or above, and have received training on this annotation rule. The professional title of the arbitration expert group shall not be lower than the intermediate professional title. The number of years they have been engaged in clinical work shall not be less than 8 years. The number of years they have been engaged in annotation shall not be less than 1 year. The assessment indicators for annotators include classification accuracy, which is required to be no less than 90%. A.1.4 Annotation tools The annotation software is self-written. Its main functions include reading, displaying, adding annotations, reviewing and modifying annotations, and saving annotation conclusions. A.1.5 Annotation environment The annotation task is carried out in a hospital's medical artificial intelligence laboratory using medical monitors and office computers, with no special environmental requirements. A.1.6 Data The data collection date is from January to July 2020.The collection device is a wearable single-lead ECG monitoring device of a certain brand, which has obtained the domestic medical device Class II registration certificate and meets the medical device standards such as YY 0885-2013.The data storage format is a binary file. The sampling rate is 200 Hz. The definition of the annotation object of this annotation task is shown in A.1.2.The location of data collection is the hyperbaric oxygen department of a certain hospital. The data source is the patients of this clinical department. There are no special requirements for the patient's age, occupation, place of origin and other characteristics. Data cleaning revolves around the effectiveness of data collection, such as the wearing condition and signal strength of the ECG monitoring device, and invalid data is manually removed. Data duplication checking mainly checks the duplication of file names, patient numbers (IDs) and file contents. For details, see the hospital's data annotate DR classification. Record the annotation results of each annotator. First, cross- annotation is used. Each color photo needs more than 2 annotators to independently annotate. If the annotation results for each disease classification are consistent, the annotation process ends. The color photo and its annotation results are included in the database. The annotation results can be sampled and reviewed as quality control. If the annotation physicians do not agree on the annotation of a single or multiple diseases during the cross-annotation stage, the color photo will be sent to the arbitration annotation stage. The arbitration annotation physician reviews the annotation results that need to be arbitrated and issues the final annotation results. During the annotation process, the consistency between the annotation physician and himself is periodically monitored. It is carried out by using the buried question verification method. For example, after completing the classification annotation of 20 diabetic retinopathy images, one of them is randomly selected for re-annotation. A.2.3 Annotators The professional title of the ophthalmologist shall not be lower than that of attending physician. The years of experience in clinical work shall not be less than 3 years. The years of experience in annotation shall not be less than 1 year. The ophthalmologist shall have received training in fundus annotation and classification. The professional title of the arbitration expert group shall not be lower than that of chief physician. The number of years of clinical work shall not be less than 10 years. The number of years of annotation work shall not be less than 3 years. Personnel assessment indicators include classification accuracy, which is required to be no less than 90%. A.2.4 Annotation tools The software used for annotation is not limited to any manufacturer. The main functions of the software include reading, displaying, adding annotations, reviewing and modifying annotations, and saving annotation conclusions of fundus image data. For details on the software interface display, please refer to the manual of the specific annotation software. A.2.5 Annotation environment The annotation task is carried out in the artificial intelligence R&D department of a hospital. Medical monitors and office computers are used. No special environment requirements are required. A.2.6 Data The data is collected from January to December 2020.The acquisition equipment is a fundus color camera of a certain brand with a field of view of 45° (with a medical device registration certificate). The data formats of fundus color photos are jpg, tiff, dcm, and ......Source: Above contents are excerpted from the full-copy PDF -- translated/reviewed by: www.ChineseStandard.net / Wayne Zheng et al. Tips & Frequently Asked Questions:Question 1: How long will the true-PDF of English version of YY/T 1833.3-2022 be delivered?Answer: The full copy PDF of English version of YY/T 1833.3-2022 can be downloaded in 9 seconds, and it will also be emailed to you in 9 seconds (double mechanisms to ensure the delivery reliably), with PDF-invoice.Question 2: Can I share the purchased PDF of YY/T 1833.3-2022_English with my colleagues?Answer: Yes. The purchased PDF of YY/T 1833.3-2022_English will be deemed to be sold to your employer/organization who actually paid for it, including your colleagues and your employer's intranet.Question 3: Does the price include tax/VAT?Answer: Yes. Our tax invoice, downloaded/delivered in 9 seconds, includes all tax/VAT and complies with 100+ countries' tax regulations (tax exempted in 100+ countries) -- See Avoidance of Double Taxation Agreements (DTAs): List of DTAs signed between Singapore and 100+ countriesQuestion 4: Do you accept my currency other than USD?Answer: Yes. www.ChineseStandard.us -- YY/T 1833.3-2022 -- Click this link and select your country/currency to pay, the exact amount in your currency will be printed on the invoice. Full PDF will also be downloaded/emailed in 9 seconds.How to buy and download a true PDF of English version of YY/T 1833.3-2022?A step-by-step guide to download PDF of YY/T 1833.3-2022_EnglishStep 1: Visit website https://www.ChineseStandard.net (Pay in USD), or https://www.ChineseStandard.us (Pay in any currencies such as Euro, KRW, JPY, AUD).Step 2: Search keyword "YY/T 1833.3-2022". Step 3: Click "Add to Cart". If multiple PDFs are required, repeat steps 2 and 3 to add up to 12 PDFs to cart. Step 4: Select payment option (Via payment agents Stripe or PayPal). Step 5: Customize Tax Invoice -- Fill up your email etc. Step 6: Click "Checkout". Step 7: Make payment by credit card, PayPal, Google Pay etc. After the payment is completed and in 9 seconds, you will receive 2 emails attached with the purchased PDFs and PDF-invoice, respectively. Step 8: Optional -- Go to download PDF. Step 9: Optional -- Click Open/Download PDF to download PDFs and invoice. See screenshots for above steps: Steps 1~3 Steps 4~6 Step 7 Step 8 Step 9 |
